pyspark 读取具有多行列的文本文件

Question

我有以下格式错误的 txt 文件：

id;text;contact_id
1;Reason contact\
\
The client was not satisfied about the quality of the product\
\
;c_102932131

我正在尝试使用 pyspark 加载文件：

df = sc.read\
.option("delimiter", ";")\
.option("header", "true")\
.option("inferSchema", "true")\
.option("multiLine", "true")\
.option("wholeFile", "true")\
.csv(os.path.join(appconfig.configs[appconfig.ENV]["ROOT_DIR"], "data", "input", file_name))

但列文本被截断，因为数据框是：

id|text|contact_id
1|Reason contact|null
null|null|c_102932131

所以我失去了所有其他的行。目标是以这种方式正确读取文件：

id|text|contact_id
1|Reason contact The client was satisfied not about the quality of the product|c_102932131

我该怎么做？ 谢谢

Answer 1

使用.wholeTextFiles然后替换new line (\n)和\最后创建df。

Example:

Spark-Scala:

sc.wholeTextFiles("<file_path>").
toDF().
selectExpr("""split(replace(regexp_replace(_2,"[\\\\|\n]",""),"id;text;contact_id",""),";") as new""").
withColumn("id",col("new")(0)).
withColumn("text",col("new")(1)).
withColumn("contact_id",col("new")(2)).
drop("new").
show(false)
//+---+---------------------------------------------------------------------------+-----------+
//|id |text                                                                       |contact_id |
//+---+---------------------------------------------------------------------------+-----------+
//|1  |Reason contactThe client was not satisfied about the quality of the product|c_102932131|
//+---+---------------------------------------------------------------------------+-----------+

Pyspark:

from pyspark.sql.functions import *

sc.wholeTextFiles("<file_path>").\
toDF().\
selectExpr("""split(replace(regexp_replace(_2,'[\\\\\\\\|\n]',''),"id;text;contact_id",""),";") as new""").\
withColumn("id",col("new")[0]).\
withColumn("text",col("new")[1]).\
withColumn("contact_id",col("new")[2]).\
drop("new").\
show(10,False)
#+---+---------------------------------------------------------------------------+-----------+
#|id |text                                                                       |contact_id |
#+---+---------------------------------------------------------------------------+-----------+
#|1  |Reason contactThe client was not satisfied about the quality of the product|c_102932131|
#+---+---------------------------------------------------------------------------+-----------+