如何阻止pyspark自动重命名重复的列

Question

我有一个包含重复列的 csv 文件。当我用 spark.read.format("CSV").load() 阅读时。它会自动重命名列名末尾附加索引值的列。

""df=spark.read.format('csv').option('header',True).load('dbfs:/FileStore/rx923b/csv/2.csv')"" 显示(df)

Here is the df

关于如何将列名设为 year, year_1 的任何想法

Answer 1

从以下链接https://dbmstutorials.com/pyspark/spark-dataframe-schema.html有解决方案：

from pyspark.sql.types import StructType # imported StructType

schema_def = StructType()  # Created a StructType object
schema_def.add("db_id","integer",True)      # Adding column 1 to StructType
schema_def.add("db_name","string",True)     # Adding column 2 to StructType
schema_def.add("db_type_cd","string",True)  # Adding column 3 to StructType

df_with_schema = spark.read.csv("file:///path_to_files/csv_file_with_duplicates.csv", schema=schema_def, header=True)

df_with_schema.printSchema()

您应该在加载文件之前创建数据集架构，这样您可以覆盖 Spark 返回的默认重复名称。