如何采样一个非常大的 CSV 文件（6GB）

Question

有一个很大的 CSV 文件（第一行作为标题），现在我想将它分成 100 个样本（例如line_num%100），如何在主内存约束下有效地做到这一点？

将文件分成 100 个较小的文件。或者每 1/100 行作为子文件 1，每 2/100 行作为子文件 2，...，每 100/100 行作为文件 100。 获取 100 个大小约为 600 M 的文件。

没有得到 100 行或 1/100 大小的样本。

我尝试这样执行：

fi  = [open('split_data//%d.csv'%i,'w') for i in range(100)]
i = 0
with open('data//train.csv') as fin:
    first = fin.readline()
    for line in fin:
        fi[i%100].write(line)
        i = i + 1
for i in range(100):
    fi[i].close()

但是文件太大，内存有限，无法运行，如何处理？ 我想打一轮~

（我的代码可以运行，但是太费时间，我误以为它崩溃了，抱歉~~）

Answer 1

按照 cmets 中的说明将文件拆分为 100 个部分（我想以模数方式将文件拆分为 100 个部分，即 range(200)-->| [0,100]; [1,101]; [ 2,102] 和 是的，将一个大文件分隔成数百个小文件)

import csv

files = [open('part_{}'.format(n), 'wb') for n in xrange(100)]
csvouts = [csv.writer(f) for f in files]
with open('yourcsv') as fin:
    csvin = csv.reader(fin)
    next(csvin, None) # Skip header
    for rowno, row in enumerate(csvin):
        csvouts[rowno % 100].writerow(row)

for f in files:
    f.close()

---

您可以islice 使用一个步骤而不是对行号进行模数来遍历文件，例如：

import csv
from itertools import islice

with open('yourcsv') as fin:
    csvin = csv.reader(fin)
    # Skip header, and then return every 100th until file ends
    for line in islice(csvin, 1, None, 100):
        # do something with line

例子：

r = xrange(1000)
res = list(islice(r, 1, None, 100))
# [1, 101, 201, 301, 401, 501, 601, 701, 801, 901]

Answer 2

根据@Jon Clements 的回答，我也会对这种变化进行基准测试：

import csv
from itertools import islice

with open('in.csv') as fin:
  first = fin.readline() # discard the header
  csvin = csv.reader( islice(fin, None, None, 100) )  # this line is the only difference
  for row in csvin:
    print row # do something with row

如果您只需要 100 个样本，您可以使用这种想法，即在文件中的等距位置进行 100 次读取。这应该适用于行长基本一致的 CSV 文件。

def sample100(path):
  with open(path) as fin:
    end = os.fstat(fin.fileno()).st_size
    fin.readline()              # skip the first line
    start = fin.tell()
    step = (end - start) / 100
    offset = start
    while offset < end:
      fin.seek(offset)
      fin.readline()            # this might not be a complete line
      if fin.tell() < end:
        yield fin.readline()    # this is a complete non-empty line
      else:
        break                   # not really necessary...
      offset = offset + step

for row in csv.reader( sample100('in.csv') ):
  # do something with row

Answer 3

我认为您可以打开同一个文件 10 次，然后独立操作（读取）每个文件，有效地将其拆分为子文件，而无需实际执行。

不幸的是，这需要提前知道文件中有多少行，并且需要读取整个内容一次以计算它们。另一方面，这应该相对较快，因为没有其他处理发生。

为了说明和测试这种方法，我创建了一个更简单的（每行只有一个项目）和小得多的 csv 测试文件，看起来像这样（第一行是标题行，不计算在内）：

line_no
1
2
3
4
5
...
9995
9996
9997
9998
9999
10000

这是代码和示例输出：

from collections import deque
import csv

# count number of rows in csv file
# (this requires reading the whole file)
file_name = 'mycsvfile.csv'
with open(file_name, 'rb') as csv_file:
    for num_rows, _ in enumerate(csv.reader(csv_file)): pass
rows_per_section = num_rows // 10

print 'number of rows: {:,d}'.format(num_rows)
print 'rows per section: {:,d}'.format(rows_per_section)

csv_files = [open(file_name, 'rb') for _ in xrange(10)]
csv_readers = [csv.reader(f) for f in csv_files]
map(next, csv_readers)  # skip header

# position each file handle at its starting position in file
for i in xrange(10):
    for j in xrange(i * rows_per_section):
        try:
            next(csv_readers[i])
        except StopIteration:
            pass

# read rows from each of the sections
for i in xrange(rows_per_section):
    # elements are one row from each section
    rows = [next(r) for r in csv_readers]
    print rows  # show what was read

# clean up
for i in xrange(10):
    csv_files[i].close()

输出：

number of rows: 10,000
rows per section: 1,000
[['1'], ['1001'], ['2001'], ['3001'], ['4001'], ['5001'], ['6001'], ['7001'], ['8001'], ['9001']]
[['2'], ['1002'], ['2002'], ['3002'], ['4002'], ['5002'], ['6002'], ['7002'], ['8002'], ['9002']]
...
[['998'], ['1998'], ['2998'], ['3998'], ['4998'], ['5998'], ['6998'], ['7998'], ['8998'], ['9998']]
[['999'], ['1999'], ['2999'], ['3999'], ['4999'], ['5999'], ['6999'], ['7999'], ['8999'], ['9999']]
[['1000'], ['2000'], ['3000'], ['4000'], ['5000'], ['6000'], ['7000'], ['8000'], ['9000'], ['10000']]