Python：难以实现读取 csv 中特定行的有效方法

Question

在我的 ML 项目中，我开始遇到 10 Gb+ 大小的 csv 文件，因此我正在尝试实施一种有效的方法来从我的 csv 文件中获取特定行。

这让我发现了itertools（据说它可以有效地跳过csv.reader 的行，而循环遍历它会加载它经过的每一行到内存中），并按照this 回答我尝试了以下：

import collections
import itertools

with open(csv_name, newline='') as f:

    ## Efficiently find total number of lines in csv
    lines = sum(1 for line in f)

    ## Proceed only if my csv has more than just its header
    if lines < 2:
        return None   
    else:

        ## Read csv file
        reader = csv.reader(f, delimiter=',')

        ## Skip to last line
        consume(reader, lines)

        ## Output last row
        last_row = list(itertools.islice(reader, None, None))

consume() 存在

def consume(iterator, n):
    "Advance the iterator n-steps ahead. If n is none, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(itertools.islice(iterator, n, n), None)

但是，我只从last_row 得到一个空列表，这意味着出了点问题。

我正在测试此代码的短 csv：

Author,Date,Text,Length,Favorites,Retweets
Random_account,2019-03-02 19:14:51,twenty-two,10,0,0

我哪里错了？

Answer 1

出了什么问题是您正在迭代文件以使其长度耗尽文件迭代器，

lines = sum(1 for line in f)

您需要重新打开文件，或使用f.seek(0)。

所以要么：

def get_last_line(csv_name):

    with open(csv_name, newline='') as f:
        ## Efficiently find total number of lines in csv
        lines = sum(1 for line in f) # the iterator is now exhausted

    if len(lines) < 2:
        return

    with open(csv_name, newline='') as f: # open file again
        # Keep going with your function
        ...

或者，

def get_last_line(csv_name):

    with open(csv_name, newline='') as f:
        ## Efficiently find total number of lines in csv
        lines = sum(1 for line in f) # the iterator is now exhausted

        if len(lines) < 2:
            return

        # we can "cheat" the iterator protocol and
        # and move the iterator back to the beginning
        f.seek(0) 
        ... # continue with the function

但是，如果你想要最后一行，你可以这样做：

for line in f:
   pass
print(line)

也许，使用collections.deque 会更快（他们在配方中使用它）：

collections.deque(f, maxlen=1)

这里有两种不同的方法来解决这个问题，让我快速创建一个文件：

Juans-MacBook-Pro:tempdata juan$ history > history.txt
Juans-MacBook-Pro:tempdata juan$ history >> history.txt
Juans-MacBook-Pro:tempdata juan$ history >> history.txt
Juans-MacBook-Pro:tempdata juan$ history >> history.txt
Juans-MacBook-Pro:tempdata juan$ cat history.txt | wc -l
    2000

好的，在 IPython repl 中：

In [1]: def get_last_line_fl(filename):
   ...:     with open(filename) as f:
   ...:         prev = None
   ...:         for line in f:
   ...:             prev = line
   ...:         if prev is None:
   ...:             return None
   ...:         else:
   ...:             return line
   ...:

In [2]: import collections
   ...: def get_last_line_dq(filename):
   ...:     with open(filename) as f:
   ...:         last_two = collections.deque(f, maxlen=2)
   ...:         if len(last_two) < 2:
   ...:             return
   ...:         else:
   ...:             return last_two[-1]
   ...:

In [3]: %timeit get_last_line_fl('history.txt')
1000 loops, best of 3: 337 µs per loop

In [4]: %timeit get_last_line_dq('history.txt')
1000 loops, best of 3: 339 µs per loop

In [5]: get_last_line_fl('history.txt')
Out[5]: '  588  history >> history.txt\n'

In [6]: get_last_line_dq('history.txt')
Out[6]: '  588  history >> history.txt\n'