如何为给定的 JSON 字符串创建 spark sql 模式？答案

【问题标题】：How to create a spark sql schema for the given JSON String?如何为给定的 JSON 字符串创建 spark sql 模式？
【发布时间】：2019-12-23 04:49:29
【问题描述】：

我正在尝试为此 JSON 创建架构。我尝试使用 schema_for_json() 然后使用 from_json() 修改列以及使用 schema_for_json() 函数后收到的架构，但它不工作。如果我尝试编写 StructType ......我会感到困惑......这么多嵌套。如果您尝试将其粘贴到 JSON 格式化程序中，它将是可读的。

{"v":[{"v":{"f":[{"v":"1"},{"v":"0"},{"v":"19"},{"v":"13"},{"v":null},{"v":"false"},{"v":null},{"v":null},{"v":null},{"v":{"f":[{"v":"/Reservations/SessionTimeout/"},{"v":"www.ahuja.com"},{"v":"Ahuja"},{"v":null},{"v":null},{"v":"/Reservations/"},{"v":"/SessionTimeout/"},{"v":"/"},{"v":""}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null}]}},{"v":null},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Orders/OrderDetail.aspx?id=c9b8b21c-fada-4f09-8e3e-74fa517d4b76"},{"v":"0"}]}},{"v":{"f":[{"v":null},{"v":"true"},{"v":null},{"v":null}]}},{"v":{"f":[{"v":"undefined"},{"v":"undefined"},{"v":null},{"v":null}]}},{"v":[]},{"v":[]},{"v":null},{"v":null},{"v":{"f":[{"v":"0"},{"v":"1"},{"v":null}]}},{"v":[]},{"v":null},{"v":[]},{"v":[{"v":{"f":[{"v":"7"},{"v":"1215153891.1572139786"}]}}]},{"v":[]},{"v":"EVENT"},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":"No"},{"v":" : "}]}},{"v":null},{"v":null},{"v":{"f":[{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":"web"},{"v":[]}]}},{"v":{"f":[{"v":"2"},{"v":"0"},{"v":"19"},{"v":"13"},{"v":null},{"v":"true"},{"v":"true"},{"v":null},{"v":null},{"v":{"f":[{"v":"/Reservations/SessionTimeout/"},{"v":"www.ahuja.com"},{"v":"Ahuja"},{"v":null},{"v":null},{"v":"/Reservations/"},{"v":"/SessionTimeout/"},{"v":"/"},{"v":""}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null}]}},{"v":null},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Orders/OrderDetail.aspx?id=c9b8b21c-fada-4f09-8e3e-74fa517d4b76"},{"v":"0"}]}},{"v":{"f":[{"v":null},{"v":"true"},{"v":null},{"v":null}]}},{"v":null},{"v":[]},{"v":[]},{"v":null},{"v":null},{"v":{"f":[{"v":"0"},{"v":"1"},{"v":null}]}},{"v":[]},{"v":null},{"v":[]},{"v":[{"v":{"f":[{"v":"1"},{"v":"41849:1313880,41853:1313893,46683:1572009,46744:1575763,48144:1639246"}]}},{"v":{"f":[{"v":"4"},{"v":"1572922686310.mste1c37"}]}},{"v":{"f":[{"v":"5"},{"v":"2019-11-04T21:13:45.537-05:00"}]}}]},{"v":[]},{"v":"PAGE"},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":"No"},{"v":" : "}]}},{"v":null},{"v":null},{"v":{"f":[{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":"web"},{"v":[]}]}}]}

以下是上面带有换行符的 JSON 的可理解格式。

{"v":[
{"v":{"f":[{"v":"1"},{"v":"0"},{"v":"19"},{"v":"13"},{"v":null},{"v":"false"},{"v":null},{"v":null},{"v":null},{"v":{"f":[{"v":"/Reservations/SessionTimeout/"},{"v":"www.ahuja.com"},{"v":"Ahuja"},{"v":null},{"v":null},{"v":"/Reservations/"},{"v":"/SessionTimeout/"},{"v":"/"},{"v":""}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null}]}},{"v":null},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Orders/OrderDetail.aspx?id=c9b8b21c-fada-4f09-8e3e-74fa517d4b76"},{"v":"0"}]}},{"v":{"f":[{"v":null},{"v":"true"},{"v":null},{"v":null}]}},{"v":{"f":[{"v":"undefined"},{"v":"undefined"},{"v":null},{"v":null}]}},{"v":[]},{"v":[]},{"v":null},{"v":null},{"v":{"f":[{"v":"0"},{"v":"1"},{"v":null}]}},{"v":[]},{"v":null},{"v":[]},{"v":[{"v":{"f":[{"v":"7"},{"v":"1215153891.1572139786"}]}}]},{"v":[]},{"v":"EVENT"},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":"No"},{"v":" : "}]}},{"v":null},{"v":null},{"v":{"f":[{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":"(entrance)"},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":"web"},{"v":[]}]}},

{"v":{"f":[{"v":"2"},{"v":"0"},{"v":"19"},{"v":"13"},{"v":null},{"v":"true"},{"v":"true"},{"v":null},{"v":null},{"v":{"f":[{"v":"/Reservations/SessionTimeout/"},{"v":"www.ahuja.com"},{"v":"Ahuja"},{"v":null},{"v":null},{"v":"/Reservations/"},{"v":"/SessionTimeout/"},{"v":"/"},{"v":""}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null}]}},{"v":null},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Reservations/SessionTimeout/"},{"v":"www.ahuja.com/Orders/OrderDetail.aspx?id=c9b8b21c-fada-4f09-8e3e-74fa517d4b76"},{"v":"0"}]}},{"v":{"f":[{"v":null},{"v":"true"},{"v":null},{"v":null}]}},{"v":null},{"v":[]},{"v":[]},{"v":null},{"v":null},{"v":{"f":[{"v":"0"},{"v":"1"},{"v":null}]}},{"v":[]},{"v":null},{"v":[]},{"v":[{"v":{"f":[{"v":"1"},{"v":"41849:1313880,41853:1313893,46683:1572009,46744:1575763,48144:1639246"}]}},{"v":{"f":[{"v":"4"},{"v":"1572922686310.mste1c37"}]}},{"v":{"f":[{"v":"5"},{"v":"2019-11-04T21:13:45.537-05:00"}]}}]},{"v":[]},{"v":"PAGE"},{"v":{"f":[{"v":null},{"v":null},{"v":null},{"v":null},{"v":"(not set)"},{"v":null},{"v":"No"},{"v":" : "}]}},{"v":null},{"v":null},{"v":{"f":[{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":"(not set)"},{"v":null},{"v":null},{"v":null},{"v":null},{"v":null}]}},{"v":"web"},{"v":[]}]}}
]}

【问题讨论】：

标签： scala apache-spark schema

【解决方案1】：

只需使用spark.read.json 读取 JSON 即可为您派生架构。我将 JSON 保存在一个文件中并按如下方式读取它 -

    val df = spark.read.json("src/main/resources/test.json")
    df.printSchema()

这会自动派生架构。我得到了以下架构，

root
 |-- v: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- v: struct (nullable = true)
 |    |    |    |-- f: array (nullable = true)
 |    |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |    |-- v: string (nullable = true)

【讨论】：