Spark数据框案例何时

Question

我正在学习 scala 的火花。我正在尝试使用某种 case 语句将一些值发布到列中。任何帮助，将不胜感激。

在输入 DF 中，我有列客户、订单、类型、消息、消息 1、消息 2。 message1 & message2 在输入 DF 中将始终为空。我想在类型为“V”时在消息 1 中发布消息，并在类型为“A”时在消息 2 中发布消息。在输出 DF 中，我应该只有一条客户记录。

DF1: 
cust, order, type, message, message1, message2
c1, o1, V, Verified, null, null
c1, o1, A, Approved, null, null
c2, o2, A, Approved, null, null
c3, o3, V, Verified, null, null

outputDF:
cust, order, type, message, message1, message2
c1, o1, A, Approved, Verified, Approved
c2, o2, A, Approved, null, Approved
c3, o3, V, Verified, Verified, null

Answer 1

如果message1 和message2 只是空值，我会使用when/otherwise 创建新列。如果message1 和message2 确实包含其他值并且您想保留它们，您可以稍微修改下面的示例并在otherwise 参数中使用现有的message1 和message2 列。

  import spark.implicits._
  import org.apache.spark.sql.functions.when
  val inputDF = spark.createDataFrame(Seq(
    ("c1", "o1", "V", "Verified", "null", "null"),
    ("c1", "o1", "A", "Approved", "null", "null"),
    ("c2", "o2", "A", "Approved", "null", "null"),
    ("c3", "o3", "V", "Verified", "null", "null")
  )).toDF("customer", "order", "type", "message", "message1", "message2")

  val newInputDF = inputDF.select("customer", "order", "type", "message")
  val outputDF = newInputDF
    .withColumn("message1", when($"type" === "V", $"message").otherwise("null"))
    .withColumn("message2", when($"type" === "A", $"message").otherwise("null"))
  outputDF.show()

Answer 2

正如其他答案中所建议的，您可以使用 when/otherwise 子句根据类型插入 message1 和 message2 值。但要满足最后一个条件，即每个客户只有一行，您可以执行以下操作：

val df = Seq(("c1", "o1", "V", "Verified", null, null),("c1", "o1", "A", "Approved", null, null), ("c2", "o2", "A", "Approved", null, null), ("c3", "o3", "V", "Verified", null, null)).toDF("cust", "order", "type", "message", "message1", "message2")

val outputDf = df.groupBy($"cust",$"order").agg(collect_list($"type").alias("type"),collect_list($"message").alias("message")).withColumn("message1", when(size($"type")===2,"Verified").when($"type"(0)==="V",$"message"(0))).withColumn("message2", when(size($"type")===2,"Approved").when($"type"(0)==="A",$"message"(0))).withColumn("message", when(size($"type")===2,lit("Approved")).otherwise($"message"(0))).withColumn("type",when(size($"type")===2,"A").otherwise($"type"(0)))

outputDf.show

给出以下输出：

+----+-----+----+--------+--------+--------+
|cust|order|type| message|message1|message2|
+----+-----+----+--------+--------+--------+
|  c2|   o2|   A|Approved|    null|Approved|
|  c1|   o1|   A|Approved|Verified|Approved|
|  c3|   o3|   V|Verified|Verified|    null|
+----+-----+----+--------+--------+--------+