使用 Python 拆分具有多个标题的 CSV 文件

Question

我有一个不断附加的 CSV 文件。它有多个标题，标题中唯一共同的地方是第一列总是“NAME”。

如何将单个 CSV 文件拆分为单独的 CSV 文件，每个标题行一个？

这是一个示例文件：

"NAME","AGE","SEX","WEIGHT","CITY"
"Bob",20,"M",120,"New York"
"Peter",33,"M",220,"Toronto"
"Mary",43,"F",130,"Miami"
"NAME","COUNTRY","SPORT","NUMBER","SPORT","NUMBER"
"Larry","USA","Football",14,"Baseball",22
"Jenny","UK","Rugby",5,"Field Hockey",11
"Jacques","Canada","Hockey",19,"Volleyball",4
"NAME","DRINK","QTY"
"Jesse","Beer",6
"Wendel","Juice",1
"Angela","Milk",3

Answer 1

鉴于其他答案，我建议的唯一修改是使用 csv.DictReader 打开。伪代码是这样的。假设文件的第一行是第一个表头

请注意，这假定条目之间没有空行或其他指示符，因此“NAME”标题出现在数据之后。如果附加文件之间有一个空行，您可以将其用作在下一行使用 infile.fieldnames() 的指示符。如果您需要将输入作为列表处理，那么前面的答案会更好。

ifile = open(filename, 'rb')
infile = cvs.Dictreader(ifile)

infields = infile.fieldnames

filenum = 1
ofile = open('outfile'+str(filenum), 'wb')
outfields = infields # This allows you to change the header field
outfile = csv.DictWriter(ofile, fieldnames=outfields, extrasaction='ignore')
outfile.writerow(dict((fn, fn) for fn in outfields))

for row in infile:
  if row['NAME'] != 'NAME':
    #process this row here and do whatever is needed
  else:
    close(ofile)
    # build infields again from this row
    infields = [row["NAME"], ...] # This assumes you know the names & order
    # Dict cannot be pulled as a list and keep the order that you want.
    filenum += 1
    ofile = open('outfile'+str(filenum), 'wb')
    outfields = infields # This allows you to change the header field
    outfile = csv.DictWriter(ofile, fieldnames=outfields, extrasaction='ignore')
    outfile.writerow(dict((fn, fn) for fn in outfields))

# This is the end of the loop. All data has been read and processed
close(ofile)
close(ifile)

如果除了第一个条目中的名称之外，新标题的确切顺序无关紧要，那么您可以按如下方式传输新列表：

infileds = [row['NAME']
for k in row.keys():
  if k != 'NAME':
    infields.append(row[k])

这将在条目 0 中创建带有 NAME 的新标题，但其他标题不会按任何特定顺序。

Answer 2

如果 csv 文件的大小不是很大——所以所有文件都可以一次在内存中——只需使用 read() 将文件读入字符串，然后在该字符串上使用正则表达式：

import re

with open(ur_csv) as f:
    data=f.read()
    chunks=re.finditer(r'(^"NAME".*?)(?=^"NAME"|\Z)',data,re.S | re.M)
    for i, chunk in enumerate(chunks, 1):
        with open('/path/{}.csv'.format(i), 'w') as fout:
            fout.write(chunk.group(1))

如果文件的大小是一个问题，您可以使用mmap 创建一个看起来像一个大字符串但不是同时在内存中的东西。

然后使用带有正则表达式的 mmap 字符串来分隔 csv 块，如下所示：

import mmap
import re

with open(ur_csv) as f:
    mf=mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    chunks=re.finditer(r'(^"NAME".*?)(?=^"NAME"|\Z)',mf,re.S | re.M)
    for i, chunk in enumerate(chunks, 1):
        with open('/path/{}.csv'.format(i), 'w') as fout:
            fout.write(chunk.group(1))

在任何一种情况下，这都会将所有块写入名为 1.csv, 2.csv 等的文件中。

Answer 3

您可以使用 python csv package 读取源文件并写入多个 csv 文件，规则是如果行中的元素 0 == "NAME"，则生成一个新文件。像这样的……

import csv

outfile_name = "out_%.csv"
out_num = 1

with open('nameslist.csv', 'rb') as csvfile:
    csvreader = csv.reader(csvfile, delimiter=',')
    csv_buffer = []

    for row in csvreader:
        if row[0] != "NAME":
            csv_buffer.append(row)
        else:
            with open(outfile_name % out_num, 'wb') as csvout:
                    for b_row in csv_buffer:
                        csvout.writerow(b_row)
                    out_num += 1
                    csv_buffer = [row]

附：我还没有实际测试过，但这是一般概念

Answer 4

每次看到标题行时，将输入复制到新的输出文件。像这样的东西（不检查错误）：

partNum = 1
outHandle = None
for line in open("yourfile.csv","r").readlines():
  if line.startswith('"NAME"'):
    if outHandle is not None:
      outHandle.close()
    outHandle = open("part%d.csv" % (partNum,), "w")
    partNum += 1
  outHandle.write(line)
outHandle.close()

如果输入不以标题行开头或输入为空，则上述内容将中断。