在 pyspark 中处理多行和回车

Question

我有一个 csv 文件，其中包含文本字段的文本限定符“”。该文件还包含在某些字段中带有回车的多行。我查看了其他 stackoverflow 帖子，但找不到可以帮助我解决我面临的问题的帖子。数据是这样的

在此处提交文件：dropbox.com/sh/nqmhdckae628ntc/AAAK7lU6wJcWkPTv5y3ZWyhRa?dl=0

"Project ID","Initiative Name",Status,Type,Submitter,"Submitted Date",CEP,Maverick,Toolbox,"External Labour",M&A,"Embedded in Plan","Start Date","End Date",Category,"TS Prime","VP Reporting Group","Benefiting BU","Vendor Name","TS Finance Prime","TS Director Prime","TS VP Prime","Finance Approved","Delivery Model",Baseline,"WLS Allocation","WLN Allocation"
1234,"xyz",Incremental,"Budget Impacting Savings","Some VP","2019-10-31 14:54:14.670",FALSE,FALSE,FALSE,FALSE,FALSE,"Embedded in plan",2019-09-09,2020-09-08,"Information Technology","some vp",Procurement,"some info some info
some info
some info
some info","something","some vp","some vp","another vp",Yes,"whatever",1.68,1,1
123,"abc",Incremental,"Non-Budget Impact Savings","some vp","2020-01-22 15:48:57.323",FALSE,FALSE,FALSE,FALSE,FALSE,,2020-01-14,2020-05-31,"Information Technology","some vp",Procurement,"some info
some info
some info",something,"some vp","some vp","some vp",,"whatever",2,0,10

这里有两条记录“项目 ID”1234 和 123。当我将此文件读入 pandas 时，它就完成了这项工作：

df_test= pd.read_csv('data.csv',sep=",",quotechar='"')

第一行的输出如下所示：

Project ID                                                         1234
Initiative Name                                                     xyz
Status                                                      Incremental
Type                                           Budget Impacting Savings
Submitter                                                       Some VP
Submitted Date                                  2019-10-31 14:54:14.670
CEP                                                               False
Maverick                                                          False
Toolbox                                                           False
External Labour                                                   False
M&A                                                               False
Embedded in Plan                                       Embedded in plan
Start Date                                                   2019-09-09
End Date                                                     2020-09-08
Category                                         Information Technology
TS Prime                                                        some vp
VP Reporting Group                                          Procurement
Benefiting BU         some info some info\r\nsome info\r\nsome info\...
Vendor Name                                                   something
TS Finance Prime                                                some vp
TS Director Prime                                               some vp
TS VP Prime                                                  another vp
Finance Approved                                                    Yes
Delivery Model                                                 whatever
Baseline                                                           1.68
WLS Allocation                                                        1
WLN Allocation                                                        1
Name: 0, dtype: objectenter code here

我们可以在输出列 Benefiting BU 中看到 /r/n。我想使用 pyspark 在 spark 中读取文件。我们正在处理的文件平均为 4.5GB。 Pandas 无法处理如此大的文件，因此使用 spark 加载和处理文件。我尝试使用“option（“multiLine”，“true”）”读取 pyspark 中的文件，但它似乎不起作用。它只给了我一条记录，那就是第一条。它无法识别第二行。

df = spark.read.format("csv").option("header", "true").option("inferSchema", "true").option("quote",'"').option("escape","/"").option("multiLine","true").csv("data.csv")
df.show()


  +----------+---------------+-----------+--------------------+---------+--------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+--------------------+--------+------------------+--------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+
|Project ID|Initiative Name|     Status|                Type|Submitter|      Submitted Date|  CEP|Maverick|Toolbox|External Labour|  M&A|Embedded in Plan|Start Date|  End Date|            Category|TS Prime|VP Reporting Group|       Benefiting BU|Vendor Name|TS Finance Prime|TS Director Prime|TS VP Prime|Finance Approved|Delivery Model|Baseline|WLS Allocation|WLN Allocation|
+----------+---------------+-----------+--------------------+---------+--------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+--------------------+--------+------------------+--------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+
|      1234|            xyz|Incremental|Budget Impacting ...|  Some VP|2019-10-31 14:54:...|false|   false|  false|          false|false|Embedded in plan|2019-09-09|2020-09-08|Information Techn...| some vp|       Procurement|some info some in...|  something|         some vp|          some vp| another vp|             Yes|      whatever|    1.68|             1|         1
123|
+----------+---------------+-----------+--------------------+---------+--------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+--------------------+--------+------------------+--------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+

注意它是如何从第二行而不是整个行中获取第一个元素并将其视为第一行的一部分。当我执行 df.count() 时，它只给了我 1，而它应该给我 2，因为我有 2 条记录。如果我在没有 multiLine 选项的情况下读取 csv，它会将每个新行识别为一行，这是公平的，因为 spark 是以这种方式构建的，但这不是我想要的。

谁能指出我正确的方向。非常感谢您的帮助。如果有人想重新创建问题，我不知道是否可以上传我在此处向您展示的测试文件。

Answer 1

好吧，我没有指定quote 和escape，但它可以工作。

df = spark.read.option("header","true").option("inferSchema","true").option("multiline","true").csv("test.csv")
df.count()

2

和

df.show(10, False)

+----------+---------------+-----------+-------------------------+---------+-----------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+----------------------+--------+------------------+-------------------------------------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+
|Project ID|Initiative Name|Status     |Type                     |Submitter|Submitted Date         |CEP  |Maverick|Toolbox|External Labour|M&A  |Embedded in Plan|Start Date|End Date  |Category              |TS Prime|VP Reporting Group|Benefiting BU                                    |Vendor Name|TS Finance Prime|TS Director Prime|TS VP Prime|Finance Approved|Delivery Model|Baseline|WLS Allocation|WLN Allocation|
+----------+---------------+-----------+-------------------------+---------+-----------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+----------------------+--------+------------------+-------------------------------------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+
|1234      |xyz            |Incremental|Budget Impacting Savings |Some VP  |2019-10-31 14:54:14.670|false|false   |false  |false          |false|Embedded in plan|2019-09-09|2020-09-08|Information Technology|some vp |Procurement       |some info some info
some info
some info
some info|something  |some vp         |some vp          |another vp |Yes             |whatever      |1.68    |1             |1             |
|123       |abc            |Incremental|Non-Budget Impact Savings|some vp  |2020-01-22 15:48:57.323|false|false   |false  |false          |false|null            |2020-01-14|2020-05-31|Information Technology|some vp |Procurement       |some info
some info
some info                    |something  |some vp         |some vp          |some vp    |null            |whatever      |2.0     |0             |10            |
+----------+---------------+-----------+-------------------------+---------+-----------------------+-----+--------+-------+---------------+-----+----------------+----------+----------+----------------------+--------+------------------+-------------------------------------------------+-----------+----------------+-----------------+-----------+----------------+--------------+--------+--------------+--------------+

表格损坏的地方，但您会看到有两行。