如何在 Spark 中对具有逗号分隔值的字符串列执行一次热编码？

Question

我有一个看起来像这样的数据框

val df = Seq(
(1,"a,b,c"),
(2,"b,c")
).toDF("id","page_path")
df.createOrReplaceTempView("df")

df.show()


+---+---------+
| id|page_path|
+---+---------+
|  1|    a,b,c|
|  2|      b,c|
+---+---------+

我想在这个 page_path 列上执行一个热编码，使得输出看起来像 -

我可以在 Spark 中使用 one-hot 编码吗？

Answer 1

可以拆分列“page_path”，然后将值分解和旋转：

 df
  .withColumn("splitted", split($"page_path",","))
  .withColumn("exploded", explode($"splitted"))
  .groupBy("id")
  .pivot("exploded")
  .count()
  // replace nulls with 0
  .na.fill(0)

输出：

+---+---+---+---+
|id |a  |b  |c  |
+---+---+---+---+
|1  |1  |1  |1  |
|2  |0  |1  |1  |
+---+---+---+---+

Answer 2

因为在你提到的问题中df.createOrReplaceTempView("df") 想到了提供与帕夏所做的相同的事情的 sql 版本。

In Databricks documenation they have mentioned many use cases with Pivot... 下面是sql爱好者的sql版本。

在这种方法中，与数据帧操作方法 pivot 使用隐式分组相反，在 sql 中不需要单独的 group by 子句。

 val df: DataFrame = Seq((1, "a,b,c"),(2, "b,c")).toDF("id", "page_path")
  df.createOrReplaceTempView("df")
  spark.sql(
    """
      |Select * from
      |( select id, explode(split( page_path ,',')) as exploded from df )
      |pivot(count(exploded) for exploded in ('is_a','is_b','is_c')
      |)
    """.stripMargin).na.fill(0).show

结果：

+---+----+----+----+
| id|is_a|is_b|is_c|
+---+----+----+----+
|  1|   0|   0|   0|
|  2|   0|   0|   0|
+---+----+----+----+

Answer 3

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object Solution {

  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf().setMaster("local[4]").setAppName("SparkClusterApp")
    val sparkSession = SparkSession.builder.config(sparkConf).getOrCreate

    import sparkSession.implicits._

    val df = Seq((1, "a,b,c"),(2, "b,c")).toDF("id", "page_path")
    df.createOrReplaceTempView("df")
    df.withColumn("_tmp", split($"page_path", "\\,")).select( $"id",
      when(array_contains($"_tmp","a"),"1").otherwise("0").as("is_a"),
      when(array_contains($"_tmp","b"),"1").otherwise("0").as("is_b"),
      when(array_contains($"_tmp","c"),"1").otherwise("0").as("is_c")).show()
  }

}