无法在 PySpark 中查询 JSON 制作的数据框列答案

【问题标题】：Cannot query JSON made dataframe column in PySpark无法在 PySpark 中查询 JSON 制作的数据框列
【发布时间】：2020-01-08 15:41:17
【问题描述】：

我有一个由 json 文件制作的数据框。架构是这样的-

>>> df.printSchema()
root
 |-- attributes: struct (nullable = true)
 |    |-- att-a: string (nullable = true)
 |    |-- att-b: string (nullable = true)
 |    |-- att-c: string (nullable = true)
 |    |-- att-d: string (nullable = true)
 |    |-- att-e: string (nullable = true)
 |    |-- att-f: string (nullable = true)
 |    |-- att-g: string (nullable = true)
 |    |-- att-h: string (nullable = true)
 |    |-- att-i: string (nullable = true)
 |    |-- att-j: string (nullable = true)
 |-- customer: string (nullable = true)

但是当我尝试访问“属性”列的内容时，我得到了这个错误 -

>>> test = spark.sql("select customer, attributes.att-a from df limit 1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u'No such struct field att in att-a, att-b, att-c, att-d, att-e, att-f, att-g, att-h, att-i, att-j; line 1 pos 17'

如果我使用反引号来引用该列，我会收到此错误-

>>> test = spark.sql("select customer, `attributes.att-a` from df limit 1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/usr/local/Cellar/apache-spark/2.4.4/libexec/python/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u"cannot resolve '`attributes.att-a`' given input columns: [df.attributes, df.customer]; line 1 pos 17;\n'GlobalLimit 1\n+- 'LocalLimit 1\n   +- 'Project [customer#7, '`attributes.att-a`]\n      +- SubqueryAlias `df`\n         +- Relation[attributes#6,customer#7] json\n"

可能是什么原因？如何查询此列？

【问题讨论】：

在指定列名时尝试使用反引号。
@blackbishop 尝试使用反引号。它对我不起作用。我在主帖中添加了错误。

标签： json apache-spark pyspark apache-spark-sql

【解决方案1】：

正如 cmets 中提到的 @blackbishop 尝试仅为 StructField 添加反引号。

test = spark.sql("select customer, attributes.`att-a` from df limit 1")

更多关于反引号的信息请参考spark java docs

【讨论】：