在 Python 的巨大列表中搜索（数字）字符串的匹配项

Question

我有一个排序字符格式的数字列表（1000 万），每个条目都是 15 个字符的恒定长度。像这样思考：

100000000000000
100000000000001
...
100000010000000

现在我想在这个列表中创建一个定期细分，看看条目是如何在不同范围内累积的。输出可能是这样的：

100000000xxxxxx, 523121 entries
100000001xxxxxx, 32231 entries

目前我已尝试将整个列表读取为一组，然后进行搜索。我已经尝试了string 和int 格式。整数版本比当前的字符串版本快 3 倍。代码如下：

collection_str = set(line.strip() for line in open(inputfile)
collection_int = set(int(line.strip()) for line in open(sys.argv[1]))

def find_str(look_for, ourset):
    count = 0
    for entry in ourset:
            if entry.startswith(look_for):
                    count += 1
    return count

def find_int(look_for, ourset):
    search_min = int(str(look_for) + "000000")
    search_max = int(str(look_for+1) + "000000")

    count = 0
    for entry in ourset:
            if entry >= search_min and entry < search_max:
                    count += 1
    return count

结果如下所示：

"int version"
100000100 27401 (0.515992sec)
100000101 0 (0.511334sec)
100000102 0 (0.510956sec)
100000103 0 (0.510467sec)
100000104 0 (0.512834sec)
100000105 0 (0.511501sec)

"string version"
100000100 27401 (1.794804sec)
100000101 0 (1.794449sec)
100000102 0 (1.802035sec)
100000103 0 (1.797590sec)
100000104 0 (1.793691sec)
100000105 0 (1.796785sec)

我想知道我是否可以以某种方式让它更快？即使使用 0,5s / 范围，如果我想经常运行它以创建一些周期性统计数据，这仍然需要时间...... 从周围的搜索中，我看到有些人使用bisect 来做类似的事情，但我似乎无法理解它应该如何工作。

Answer 1

如果列表已排序，则 bisect 将使用 bisection search 查找符合您条件的索引。看起来 bisect 比使用 numpy 数组要快得多。

import numpy as np
import bisect
from random import randint
from timeit import Timer

ip = ['1{0:014d}'.format(randint(0, 10000000)) for x in xrange(10000000)]
ip = sorted(ip)
print bisect.bisect(ip, '100000000010000')
# 9869
t = Timer("bisect.bisect(ip, '100000000010000')", 'from __main__ import bisect, ip')
print t.timeit(100)
# 0.000268309933485 seconds

ip_int = map(int, ip)
print bisect.bisect(ip_int, 100000000010000)
# 9869
t = Timer("bisect.bisect(ip_int, 100000000010000)", 'from __main__ import bisect, ip_int')
print t.timeit(100)
# 0.000137443078672 seconds

ip_numpy = np.array(ip_int)
print np.sum(ip_numpy <= 100000000010000)
# 9869
t = Timer("np.sum(ip_numpy <= 100000000010000)", 'from __main__ import np, ip_numpy')
print t.timeit(100)
# 8.23690123071 seconds

Binary search algorithm

Answer 2

将其放入一个 numpy 数组中。然后，您可以使用既好又快的矢量化 :)

from random import randint
import numpy
ip = numpy.array(['1{0:014d}'.format(randint(0, 10000000)) for x in xrange(10000000)], dtype=numpy.int64)

numpy.sum(ip <= 100000000010000)
# 9960
%timeit numpy.sum(ip <= 100000000010000)
# 10 loops, best of 3: 35 ms per loop

将其放在您的搜索功能方面：

import numpy

def find_numpy(look_for, ourset):
    search_min = int('{0:0<15s}'.format(str(look_for)))
    search_max = int('{0:0<15s}'.format(str(look_for+1)))
    return numpy.sum((ourset >= search_min) & (ourset < search_max))

with open('path/to/your/file.txt', 'r') as f:
    ip = numpy.array([line.strip() for line in f], dtype=numpy.int64)

find_numpy(1000000001, ip)
# 99686
%timeit find_numpy(1000000001, ip)
# 10 loops, best of 3: 86.6 ms per loop