从 HDFS 和 Schema 问题读取 Parquet

Question

当我尝试从 HDFS 读取镶木地板文件时，我得到了所有混合大小写的架构。有什么方法可以将其转换为全部小写？

df=spark.read.parquet(hdfs_location)

df.printSchema();
root
|-- RecordType: string (nullable = true)
|-- InvestmtAccnt: string (nullable = true)
|-- InvestmentAccntId: string (nullable = true)
|-- FinanceSummaryID: string (nullable = true)
|-- BusinDate: string (nullable = true)

What i need is like below


root
|-- recordtype: string (nullable = true)
|-- investmtaccnt: string (nullable = true)
|-- investmentaccntid: string (nullable = true)
|-- financesummaryid: string (nullable = true)
|-- busindate: string (nullable = true)

Answer 1

首先读取拼花文件

df=spark.read.parquet(hdfs_location)

然后使用.toDF 函数创建包含所有lower column names 的数据框

df=df.toDF(*[c.lower() for c in df.columns])