读取用双引号括起来但带有换行符的csv文件

Question

我有csv，列中有换行符。以下是我的示例：

"A","B","C"
1,"This is csv with 
newline","This is another column"
"This is newline
and another line","apple","cat"

我可以在 spark 中读取文件，但列内的换行符被视为单独的行。

如何将其准备为 csv，并将文本包含在双引号内。

我只使用 apache csv 插件和 apache 读取文件。

alarms = sc.textFile("D:\Dataset\oneday\oneday.csv")

这给了我 RDD ：

**example.take(5)**

[u'A,B,C', u'1,"This is csv with ', u'newline",This is another column', u'"This is newline', u'and another line",apple,cat']

Spark 版本：1.4

Answer 1

标准 python 库中的 csv 模块开箱即用：

>>> txt = '''"A","B","C"
1,"This is csv with 
newline","This is another column"
"This is newline
and another line","apple","cat"'''
>>> import csv
>>> import io
>>> with io.BytesIO(txt) as fd:
    rd = csv.reader(fd)
    for row in rd:
        print row


['A', 'B', 'C']
['1', 'This is csv with \nnewline', 'This is another column']
['This is newline\nand another line', 'apple', 'cat']

这可以与binaryFiles 一起使用（与textFile 相比有显着的性能损失）：

>>> (sc.binaryFiles(path)
        .values()
        .flatMap(lambda x: csv.reader(io.BytesIO(x))))

Answer 2

您不需要导入任何内容。下面提出的解决方案创建第二个文件仅用于演示目的。您可以在修改后读取该行，而无需将其写入任何地方。

with open(r'C:\Users\evkouni\Desktop\test_in.csv', 'r') as fin:
    with open(r'C:\Users\evkouni\Desktop\test_out.csv', 'w') as fout:
        cont = fin.readlines()
        for line in cont[:-1]:
            if line.count('"') % 2 == 1 and '"\n' not in line:
                line = line.replace('\n', '')
            fout.write(line)

#DEMO

#test_in.csv
#------------
#"A";"B";"C"
#1;"This is csv with 
#newline";"This is another column"
#"This is newline

#test_out.csv
#------------
#"A";"B";"C"
#1;"This is csv with newline";"This is another column"
#"This is newline

如果您有不清楚的地方，请告诉我。

Answer 3

如果您想使用换行符从 csv 创建数据帧并用双引号引用而不重新发明*，请使用 spark-csv 和 common-csv 库：

from pyspark.sql import SQLContext
df = sqlContext.load(header="true",source="com.databricks.spark.csv", path = "hdfs://analytics.com.np:8020/hdp/badcsv.csv")