难以转换为 Spark 数据帧的 JSON答案

【问题标题】：Difficult JSON To Convert To A Spark Dataframe难以转换为 Spark 数据帧的 JSON
【发布时间】：2021-06-28 12:12:05
【问题描述】：

我一直在尝试将 JSON 加载到 pyspark 数据帧中，但我在这里有点挣扎。

这是我迄今为止尝试过的（有和没有多行）：

import json
newJson = json.dumps(testjson)
newdf = spark.read.json(sc.parallelize([newJson]))
newdf.display()

JSON 文件：

testjson = [
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
 ('{"id":434, "address" : ["432.432.432.432", "432.432.432.432", "432.432.432.432", "432.432.432.432"]}',), 
]

尝试显示数据框时，我得到“corrupt_record”。我做错了什么？

【问题讨论】：

JSON 似乎无效。尝试通过验证器运行它。

标签： json apache-spark pyspark apache-spark-sql

【解决方案1】：

尝试将其转换为字符串列表。 Spark 无法理解字符串元组的列表。 json.dumps 也是不必要的，因为 Spark 应该能够理解您的 json 输入。

df = spark.read.json(sc.parallelize([i[0] for i in testjson]))

df.show(truncate=False)
+--------------------------------------------------------------------+---+
|address                                                             |id |
+--------------------------------------------------------------------+---+
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
|[432.432.432.432, 432.432.432.432, 432.432.432.432, 432.432.432.432]|434|
+--------------------------------------------------------------------+---+

【讨论】：