根据spark scala中完整df的列值设置列值答案

【问题标题】：Setting the column value based on the column value of complete df in spark scala根据spark scala中完整df的列值设置列值
【发布时间】：2021-07-13 23:36:14
【问题描述】：

可以帮助我解决用例：下面是数据集

+-----------+-------------+------------+
|artistId   |musicalGroups|displayName |
+-----------+-------------+------------+
|wa_16      |wa_31        |Exods       |
|wa_38      |wa_16        |Kirk        |
+-----------+-------------+------------+

我想根据 musicalGroups 值填充列 name，并根据 artistId 设置 name strong> displayName 列值。就像在下面的示例中，我们将 wa_16 作为名为 Exods 的艺术家 ID，因此 name 列应该具有根据它的艺术家 ID 的 displayName。示例：

+-----------+-------------+------------+
|artistId   |musicalGroups|displayName |name
+-----------+-------------+------------+
|wa_16      |wa_31        |Exods       |null
|wa_38      |wa_16        |Kirk        |Exods
+-----------+-------------+------------+

Tried via self join on artistId and musicalGroups, but it was not working.
Can some help me to solve this usecase?

【问题讨论】：

请检查并告诉我们以下代码是否适合您

标签： scala apache-spark apache-spark-sql

【解决方案1】：

val df = `your existing dataframe`

// Derive new dataset from the original dataset
val newDF = df.select("artistId", "displayName").distinct()

// Join new dataset with original dataset based on the common key and select the relevant columns
val combinedDF = df.join(newDF, df.col("musicalGroups") === newDF.col("artistId"), "leftOuter").select(df.col("artistId") as "artistId", df.col("musicalGroups") as "musicalGroups", df.col("displayName") as "displayName", newDF.col("displayName") as "name")

【讨论】：

这样设置错误值。在这里，它根据艺术家 ID 设置值，但我们需要在完整的 df 中搜索 musicGroup 值，然后将值设置为具有艺术家 ID 名称的行。
通过上述连接，您可以获得所有行的名称，基于musicalGroup 和它的相对artistId 组合。

【解决方案2】：

IIUC，您可以使用 pivot() 和 groupBy()

df = spark.createDataFrame([("wa_16","wa_31","Exods"),("wa_38","wa_16","Krik")],["artistId","musicalGroups","displayName"])
df_grp = df.groupBy("artistId", "musicalGroups", "displayName").pivot("displayName").agg(F.first(F.col("artistId")))
df.show()
df_grp.show()

+--------+-------------+-----------+
|artistId|musicalGroups|displayName|
+--------+-------------+-----------+
|   wa_16|        wa_31|      Exods|
|   wa_38|        wa_16|       Krik|
+--------+-------------+-----------+

+--------+-------------+-----------+-----+-----+
|artistId|musicalGroups|displayName|Exods| Krik|
+--------+-------------+-----------+-----+-----+
|   wa_16|        wa_31|      Exods|wa_16| null|
|   wa_38|        wa_16|       Krik| null|wa_38|
+--------+-------------+-----------+-----+-----+

【讨论】：