如何在 Spark 中更新数据框列

Question

我有一个数据框，其中有 2 个 json 列。我需要根据 j1 列更新 j2 列。如果 j2 列的元素为空，则从 j1 列中选择元素值。 j1 为 JSON 字符串，j2 为 JSON 数组。

输入数据帧，

+---------------------------+---------------------------------------+
|         j1                |              j2                       |
+---------------------------+---------------------------------------+
|{"A": "50", "B": "30"}     | [{"A": null}, {"A": "20", "B": null}] |
+---------------------------+---------------------------------------+

输出

+------------------------+---------------------------------------+
|          j1            |              j2                       |
+------------------------+---------------------------------------+
|{"A": "50", "B": "30"}  | [{"A":"50"}, {"A": "20", "B": "30"}]  |
+------------------------+---------------------------------------+

spark 版本：2.4.0 语言：scala

Answer 1

您可以使用 regexp_replace 函数来获得所需的结果。

  val spark = SparkSession.builder().master("local[*]").getOrCreate()
  import spark.implicits._
  spark.sparkContext.setLogLevel("ERROR")

  val inDF = Seq(("""{"A": "50", "B": "30"} """, """[{"A": null}, {"A": "20", "B": null}]""")).toDF("j1", "j2")

  inDF.withColumn("A", get_json_object('j1, "$.A"))
    .withColumn("B", get_json_object('j1, "$.B"))
    .withColumn("j2", regexp_replace('j2, lit("\"A\": null"), 
      concat(lit("\"A\": "), lit("\""), 'A, lit("\""))))
    .withColumn("j2", regexp_replace('j2, lit("\"B\": null"),
      concat(lit("\"B\": "), lit("\""), 'B, lit("\""))))
    .drop("A", "B")
    .show(false)

+-----------------------+-------------------------------------+
|j1                     |j2                                   |
+-----------------------+-------------------------------------+
|{"A": "50", "B": "30"} |[{"A": "50"}, {"A": "20", "B": "30"}]|
+-----------------------+-------------------------------------+