通过在 spark 中使用 scala 加载 csv 文件来创建数据框

Question

但是 csv 文件添加了额外的双引号，这导致所有 cloumns 成为单列

有四列，标题和两行

"""SlNo"",""Name"",""Age"",""contact"""
"1,""Priya"",78,""Phone"""
"2,""Jhon"",20,""mail"""

val df = sqlContext.read.format("com.databricks.spark.csv").option("header","true").option("delimiter",",").option("inferSchema","true").load ("bank.csv") 
df: org.apache.spark.sql.DataFrame = ["SlNo","Name","Age","contact": string]

Answer 1

您可以做的是使用sparkContext 读取它并将所有 " 替换为empty 并使用zipWithIndex() 分隔标题和文本数据，以便 custom schema 和 row rdd 数据。最后只需在 sqlContext 的 createDataFrame api 中使用 row rdd 和 schema

//reading text file, replacing and splitting and finally zipping with index
val rdd = sc.textFile("bank.csv").map(_.replaceAll("\"", "").split(",")).zipWithIndex()
//separating header to form schema
val header = rdd.filter(_._2 == 0).flatMap(_._1).collect()
val schema = StructType(header.map(StructField(_, StringType, true)))
//separating data to form row rdd
val rddData = rdd.filter(_._2 > 0).map(x => Row.fromSeq(x._1))
//creating the dataframe
sqlContext.createDataFrame(rddData, schema).show(false)

你应该得到

+----+-----+---+-------+
|SlNo|Name |Age|contact|
+----+-----+---+-------+
|1   |Priya|78 |Phone  |
|2   |Jhon |20 |mail   |
+----+-----+---+-------+

希望回答对你有帮助