pyspark：重新排列结构序列的嵌套数组

Question

我有一个这种格式的数据框，我想重新排列 item 列中的字段。

root
 |-- order: string (nullable = true)
 |-- dt: struct (nullable = true)
 |-- item: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a: long (nullable = true)
 |    |    |-- b: string (nullable = true)
 |    |    |-- c: long (nullable = true)

所以这是我正在寻找的所需格式。

root
 |-- order: string (nullable = true)
 |-- dt: struct (nullable = true)
 |-- item: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- a: long (nullable = true)
 |    |    |-- c: string (nullable = true)
 |    |    |-- b: long (nullable = true)

Answer 1

你可以使用transform函数：

from pyspark.sql import functions as F

result = df.withColumn(
    "item",
    F.expr("transform(item, x -> struct(x.a as a, x.c as c, x.b as b))")
)