我正在使用 apache spark 来解析 json 文件。如何从 json 文件中获取嵌套键，无论它是数组还是嵌套键答案

【问题标题】：I am using apache spark to parse json files . How to get nested key from json files wheather it's array or nested key我正在使用 apache spark 来解析 json 文件。如何从 json 文件中获取嵌套键，无论它是数组还是嵌套键
【发布时间】：2017-09-22 12:55:14
【问题描述】：

我有多个 json 文件来保持 json 数据初始化。 Json 结构如下所示。

{ 
  "Name":"Vipin Suman",
  "Email":"vpn2330@gmail.com",
 "Designation":"Trainee Programmer",
 "Age":22 ,
 "location":
    {"City":
           {
            "Pin":324009,
            "City Name":"Ahmedabad"
           },
    "State":"Gujarat"
   },
 "Company":
          {
           "Company Name":"Elegant",
           "Domain":"Java"
          }, 
 "Test":["Test1","Test2"]

}

我试过了

    String jsonFilePath = "/home/vipin/workspace/Smarten/jsonParsing/Employee/Employee-03.json";

    String[] jsonFiles = jsonFilePath.split(",");

    Dataset<Row> people = sparkSession.read().json(jsonFiles);

我正在为此获取架构

root
 |-- Age: long (nullable = true)
 |-- Company: struct (nullable = true)
 |    |-- Company Name: string (nullable = true)
 |    |-- Domain: string (nullable = true)
 |-- Designation: string (nullable = true)
 |-- Email: string (nullable = true)
 |-- Name: string (nullable = true)
 |-- Test: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- location: struct (nullable = true)
 |    |-- City: struct (nullable = true)
 |    |    |-- City Name: string (nullable = true)
 |    |    |-- Pin: long (nullable = true)
 |    |-- State: string (nullable = true)

我正在查看表格：-

    +---+--------------+------------------+-----------------+-----------+--------------+--------------------+
|Age|       Company|       Designation|            Email|       Name|          Test|            location|
+---+--------------+------------------+-----------------+-----------+--------------+--------------------+
| 22|[Elegant,Java]|Trainee Programmer|vpn2330@gmail.com|Vipin Suman|[Test1, Test2]|[[Ahmedabad,32400...|
+---+--------------+------------------+-----------------+-----------+--------------+--------------------+

我希望结果为：-

            Age   |  Company Name    | Domain|  Designation |  Email           |    Name          |  Test   | City Name      |  Pin   |   State   |

           22     | Elegant MicroWeb | Java  |  Programmer  | vpn2330@gmail.com | Vipin Suman     | Test1  |  Ahmedabad      | 324009  | Gujarat 
           22     | Elegant MicroWeb | Java  |  Programmer  | vpn2330@gmail.com | Vipin Suman     | Test2  |  Ahmedabad      | 324009  |

我如何在上面的表格中获得表格。我尝试了一切。我是 apache spark 的新手，谁能帮帮我？？

【问题讨论】：

标签： java json apache-spark apache-spark-sql

【解决方案1】：

在 Scala 中可以这样做：

people.select(
  $"Age",
  $"Company.*",
  $"Designation",
  $"Email",
  $"Name",
  explode($"Test"),
  $"location.City.*",
  $"location.State")

很遗憾，Java 中的以下代码会失败：

people.select(
  people.col("Age"),
  people.col("Company.*"),
  people.col("Designation"),
  people.col("Email"),
  people.col("Name"),
  explode(people.col("Test")),
  people.col("location.City.*"),
  people.col("location.State"));

您可以改用selectExpr：

people.selectExpr(
  "Age",
  "Company.*",
  "Designation",
  "Email",
  "Name",
  "EXPLODE(Test) AS Test",
  "location.City.*",
  "location.State");

PS： 您可以将路径传递到一个或多个目录，而不是 sparkSession.read().json(jsonFiles); 中的 JSON 文件列表。

【讨论】：

【解决方案2】：

我建议你在 scala 中做你的工作，这是由 spark 更好地支持的。做你的工作，你可以使用"select" API来选择一个特定的列，使用别名来重命名一个列，你可以参考这里说如何选择复杂的数据格式(https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html)

根据您的结果，您还需要使用“explode”API (Flattening Rows in Spark)

【讨论】：