Pyspark 转换复杂的数据格式

Question

用 spark 读取 parquet 文件

df = spark.read.parquet("path_to_file")
df.show(2)

我的 df 包含

**Output**

+------+-----------------+
| col1 |       col2      |
+------+-----------------+
| "A1" |  {"x":1,"y":2}  |
+------+-----------------+
| "A2" |  {"z":3}        |
+------+-----------------+

我想将数据框转换为喜欢

+------+------+------+------+
| col1 |  x   |  y   |  z   |
+------+------+------+------+
| "A1" |  1   |  2   | Null |
+------+------+------+------+
| "A2" | Null | Null |  3   |
+------+------+------+------+

初始数据框的模式显示

DataFrame[col1: string, col2: string]

我在 Windows 7 机器上使用 pyspark 2.3.2

Answer 1

这个解决方案并不完全符合您的要求，但也许您不这么认为。因此，如果不适合您，请添加 cmets。

> from pyspark.sql import functions as F
> from pyspark.sql.types import *

> df.show()
+----+-------------+
|col1|         col2|
+----+-------------+
|  A1|{"x":1,"y":2}|
|  A2|      {"z":3}|
+----+-------------+

> df.printSchema()
root
 |-- col1: string (nullable = true)
 |-- col2: string (nullable = true)

> df = df.withColumn(
..  "col2",
..  F.from_json("col2", MapType(StringType(),IntegerType()))
..)

> df.show()
+----+----------------+
|col1|            col2|
+----+----------------+
|  A1|[x -> 1, y -> 2]|
|  A2|        [z -> 3]|
+----+----------------+

> df.select(
..    "col1",
..    "col2.x",
..    "col2.y",
..    "col2.z",
..).show()
+----+----+----+----+
|col1|   x|   y|   z|
+----+----+----+----+
|  A1|   1|   2|null|
|  A2|null|null|   3|
+----+----+----+----+