聚合酶。在 Spark 中使用 filter 和 groupby答案

【问题标题】：Agg. with filter and groupby in Spark聚合酶。在 Spark 中使用 filter 和 groupby
【发布时间】：2021-04-25 00:15:18
【问题描述】：

我正在根据groupBy 条件进行聚合，并在我现有的 Spark/Scala DataFrame 上应用一些过滤器。但是在执行我的代码时，我得到'无法解析'flag'给定输入列：'

val someDF = Seq(
 (1, 111,100,100,"C","5th","Y",11),
 (1, 111,100,100,"C","5th","Y",11),
 (2, 222,200,200,"C","5th","Y",22),
 (2, 222,200,200,"C","5th","Y",22)
 ).toDF("id","rollno","sub1","sub2","flag","class","status","sno")

var df2 = someDF.groupBy("id","rollno")
.agg(sum("sub1").alias("sub1"),sum("sub2").alias("sub2"))
.filter(col("flag") === "C")
.filter(length(col("rollno")) >= 2)
.filter(col("class") === ("5th") || col("class") === ("6th"))
.filter(substring(col("rollno"), 1, 2) === col("sno"))
.filter(col("status") === "Y")
.select("id", "rollno", "sub1", "sub2", "flag", "class", "sno", "status")

错误：

org.apache.spark.sql.AnalysisException: 无法解析 'flag' 给定的输入列：[id, rollno, sub1, sub2];; '过滤器（'标志= C）

预期结果：

+---+------+----+----+----+-----+------+---+
| id|rollno|sub1|sub2|flag|class|status|sno|
+---+------+----+----+----+-----+------+---+
|  1|   111| 200| 200|   C|  5th|     Y| 11|
|  2|   222| 400| 400|   C|  5th|     Y| 22|
+---+------+----+----+----+-----+------+---+

【问题讨论】：

在分组前应用过滤器

标签： sql apache-spark pyspark apache-spark-sql

【解决方案1】：

聚合后其他列已消失，因此您无法根据这些列进行过滤。您需要在分组依据之前进行过滤。如果要保留它们，还需要按其他列分组。

var df2 = someDF
   .filter(col("flag") === "C")
   .filter(length(col("rollno")) >= 2)
   .filter(col("class") === ("5th") || col("class") === ("6th"))
   .filter(substring(col("rollno"), 1, 2) === col("sno"))
   .filter(col("status") === "Y")
   .groupBy("id", "rollno", "flag", "class", "sno", "status")
   .agg(sum("sub1").alias("sub1"),sum("sub2").alias("sub2"))
   .select("id", "rollno", "sub1", "sub2", "flag", "class", "sno", "status")

df2.show
+---+------+----+----+----+-----+---+------+
| id|rollno|sub1|sub2|flag|class|sno|status|
+---+------+----+----+----+-----+---+------+
|  1|   111| 200| 200|   C|  5th| 11|     Y|
|  2|   222| 400| 400|   C|  5th| 22|     Y|
+---+------+----+----+----+-----+---+------+

【讨论】：