Spark DataFrame 选择空值

Question

我有一个 spark 数据框，其中几列为空。我需要创建一个新的数据框，添加一个新列“error_desc”，该列将提及每行具有空值的所有列。我需要在不提及每个列名的情况下动态执行此操作。

例如：如果我的数据框低于

+-----+------+------+
|Rowid|Record|Value |
+-----+------+------+
|    1|     a|     b|
|    2|  null|     d|
|    3|     m|  null|
+-----+------+------+

我的最终数据框应该是

+-----+------+-----+--------------+
|Rowid|Record|Value|    error_desc|
+-----+------+-----+--------------+
|    1|     a|    b|          null|
|    2|  null|    d|record is null|
|    3|     m| null| value is null|
+-----+------+-----+--------------+

Answer 1

我在 Input DataFrame 中添加了几行以涵盖更多情况。您不需要对任何列进行硬编码。在UDF下方使用，它会给你想要的输出。

scala> import org.apache.spark.sql.Row

scala> import org.apache.spark.sql.expressions.UserDefinedFunction

scala> df.show()
+-----+------+-----+
|Rowid|Record|Value|
+-----+------+-----+
|    1|     a|    b|
|    2|  null|    d|
|    3|     m| null|
|    4|  null|    d|
|    5|  null| null|
| null|     e| null|
|    7|     e|    r|
+-----+------+-----+


scala> def CheckNull:UserDefinedFunction = udf((Column:String,r:Row) => {
 |     var check:String = ""
 |     val ColList = Column.split(",").toList
 |     ColList.foreach{ x =>
 |       if (r.getAs(x) == null)
 |       {
 |         check = check + x.toString + " is null. "
 |       }}
 |   check
 |   })


scala> df.withColumn("error_desc",CheckNull(lit(df.columns.mkString(",")),struct(df.columns map col: _*))).show(false)
+-----+------+-----+-------------------------------+
|Rowid|Record|Value|error_desc                     |
+-----+------+-----+-------------------------------+
|1    |a     |b    |                               |
|2    |null  |d    |Record is null.                |
|3    |m     |null |Value is null.                 |
|4    |null  |d    |Record is null.                |
|5    |null  |null |Record is null. Value is null. |
|null |e     |null |Rowid is null. Value is null.  |
|7    |e     |r    |                               |
+-----+------+-----+-------------------------------+