我只需将元组列表拆分为 N 个子列表，其中 N 是我计算机上的处理器数量，创建一个大小为 N 的多处理池，然后并行搜索 N 个子列表。我还将使用集合 search_list 初始化池中的每个进程，以便测试子列表的元素是否是所寻找的元素之一成为 O(1) 操作（如果 search_list 实际上大于2 个元素）。 然后worker函数search_the_tuple将传入的列表转换为集合，并与search_list进行集合交集并返回结果。一旦第一个并行进程发现非零长度集合返回，终止剩余进程。
问题是，将子列表转换为集合并执行交集（由库例程完成）是否会比循环遍历子列表和测试集合成员（主要是 Python 字节码）快得多.
import multiprocessing as mp
def init_pool(s_list):
global search_list
search_list = set(s_list)
def search_the_tuple(t_sub_lib):
for item in t_sub_lib:
if item in search_list:
return(item)
def split(lst, n):
# a generator expression to split the library of tuples into n (narly) equal sub lists:
k, m = divmod(len(lst), n)
return (lst[i * k + min(i, m):(i + 1) * k + min(i + 1, m)] for i in range(n))
def main():
search_list = [('a','a','1'), ('m', '1', 'l')]
tuple_library = [
('g', 'z', '1'), ('h', '3', 'b'), ('i', 'a', 'l'), ('j', 'z', '1'), ('k', '3', 'b'), ('l', '1', 'l'),
('m', 'z', '1'), ('n', '3', 'b'), ('o', '1', 'l'), ('p', 'z', '1'), ('q', '3', 'b'), ('r', '1', 'l'),
('a', 'z', '1'), ('r', '3', 'b'), ('m', '1', 'l'), ('d', 'z', '1'), ('e', '3', 'b'), ('f', '1', 'l')
]
n_processors = mp.cpu_count()
with mp.Pool(n_processors, initializer=init_pool, initargs=(search_list,)) as pool:
t_sub_libs = split(tuple_library, n_processors)
for t in pool.imap_unordered(search_the_tuple, t_sub_libs):
if t is not None:
print(t)
break
# When we exit this block, terminate will be called on pool and
# all processes will be killed.
# required for Windows:
if __name__ == '__main__':
main()
打印：
('m', '1', 'l')
更新
我做了一个小基准测试，看看是否值得将列表转换为集合，然后执行交集。在这个基准测试中，我有一个包含 20,000 个元素的列表，并且为了让转换为集合具有优势，我正在寻找的元素位于最后：
import timeit
s = set([(19999, 19999), (21000, 21000)])
lst = [(i,i) for i in range(20000)]
def s1():
for i in lst:
if i in s:
return i
def s2():
s2 = set(lst)
return s & s2
print(timeit.timeit('s1()', number=1000, globals=globals()))
print(timeit.timeit('s2()', number=1000, globals=globals()))
打印：
0.9409163000000262
1.8975496000000476
但即使在这里，set-conversion 版本s2 的速度也几乎是原来的两倍。但是，当我要查找的值位于列表中间时（平均而言），我们有：
import timeit
s = set([(10000, 10000), (21000, 21000)])
lst = [(i,i) for i in range(20000)]
def s1():
for i in lst:
if i in s:
return i
def s2():
s2 = set(lst)
return s & s2
print(timeit.timeit('s1()', number=1000, globals=globals()))
print(timeit.timeit('s2()', number=1000, globals=globals()))
打印：
0.5094996000000265
1.8004526000001988
而且，正如您所料，s1 现在的运行速度几乎是原来的两倍。
结论
- 您绝对应该将
search_list 转换为set，因为它会被多次搜索。这可能是您唯一可用的优化。
- 您仅指定
tuple_library 非常大。这是相当不具体的。我的基准建议将您的tuple_library 转换为集合是不有利的，因为它只被搜索一次。
- 虽然，我已经展示了如何使用多处理“分而治之”，但我并不清楚（再次不知道您的
tuple_library 实际上有多大，甚至可能无关紧要）多处理将完全提高性能（它甚至可能会损害性能）。仅创建进程池会产生开销，然后将子列表传递给“生活”在不同地址空间中的进程会再次产生开销。您只需尝试一下，看看是否有帮助。
更新 2
@lllrnr101 所做的基准测试存在问题。我会提出这个作为评论，但不幸的是，这对于这样的人来说太长了。首先，函数normal_search_multiple_tuples_in_tuples_list应该编码为：
def normal_search_multiple_tuples_in_tuples_list(tuples_to_search, big_tuple_list):
s = set(tuples_to_search)
for t in big_tuple_list:
if t in s:
return t
请注意，在 OP 的问题中，OP 在找到第一个匹配项后返回，并且 不 尝试查找所有匹配项。至少将tuples_to_search 转换为一个集合可能更有效，因为它将被搜索多次。
其次，真正需要比较的只有两个策略是normal_search_multiple_tuples_in_tuples_list 和intersect_search_tuples_in_tuples_list。后面的基准测试涉及首先将列表转换为集合并搜索它，没有正确考虑转换成本，并且不对应实际问题（将列表转换为集合打印的时间是单个成本转换，而其他时间打印为 100000 次迭代）。
第三，OP 指定有问题的列表非常大，不管这意味着什么，但我认为它大于 10 和 100，可能还有 1000。所以应该针对更大的列表，timeit 进行更少的迭代，以便在合理的时间内完成。这导致以下行为：
import timeit
import time
import random
import string
from random import randint
from random import sample
data_values = string.ascii_lowercase + string.digits
len_data_values = len(data_values)
random_lower_limit = 0
random_upper_limit = len_data_values - 1
def normal_search_multiple_tuples_in_tuples_list(tuples_to_search, big_tuple_list):
s = set(tuples_to_search)
for t in big_tuple_list:
if t in s:
return t
def intersect_search_tuples_in_tuples_list(tuples_to_search, big_tuple_list):
tuples_to_search_set = set(tuples_to_search)
big_tuple_set = set(big_tuple_list)
return tuples_to_search_set.intersection(big_tuple_set)
def get_big_data(big_tuple_list_size):
big_tuple_list = [(
data_values[randint(random_lower_limit, random_upper_limit)],
data_values[randint(random_lower_limit, random_upper_limit)],
data_values[randint(random_lower_limit, random_upper_limit)]
) for _ in range(big_tuple_list_size)]
return big_tuple_list
def get_small_data(big_tuple_list, tuples_to_search_size):
return sample(big_tuple_list, tuples_to_search_size)
def get_required_data(big_tuple_list_size, tuples_to_search_size):
big_tuple_list = get_big_data(big_tuple_list_size)
tuples_to_search = get_small_data(big_tuple_list, tuples_to_search_size)
return (big_tuple_list, tuples_to_search)
if __name__ == '__main__':
test_combinations = [(10000, 5), (20000, 5), (100000, 5)]
functions_to_test = [
normal_search_multiple_tuples_in_tuples_list,
intersect_search_tuples_in_tuples_list
]
for big_tuple_list_size, tuples_to_search_size in test_combinations:
tuples_to_search, big_tuple_list = get_required_data(big_tuple_list_size, tuples_to_search_size)
print(f'For a run of searching {tuples_to_search_size} needles in {big_tuple_list_size} size haystack')
for func in functions_to_test:
print(f'''Time taken by {func.__name__}: {timeit.timeit('func(tuples_to_search, big_tuple_list)', number=1000, globals=globals())}''')
print()
打印：
For a run of searching 5 needles in 10000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 0.681931
Time taken by intersect_search_tuples_in_tuples_list: 0.5806513
For a run of searching 5 needles in 20000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 0.8495027999999998
Time taken by intersect_search_tuples_in_tuples_list: 0.8799418999999999
For a run of searching 5 needles in 100000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 8.51018
Time taken by intersect_search_tuples_in_tuples_list: 8.115366799999999
时间上没有太大的区别。我自己的基准测试表明，其中一个寻找的项目是列表的中间元素，这表明不转换列表的性能明显优于将列表转换为集合。 但是这个基准和我的最大区别在于我的基准不会将tuples_to_search 列表转换为集合；这个集合是预先创建的，因为 OP 知道他们在寻找什么。 如果我们从我们拥有的基准中删除创建该集合的成本：
import timeit
import time
import random
import string
from random import randint
from random import sample
data_values = string.ascii_lowercase + string.digits
len_data_values = len(data_values)
random_lower_limit = 0
random_upper_limit = len_data_values - 1
def normal_search_multiple_tuples_in_tuples_list(tuples_to_search, big_tuple_list):
for t in big_tuple_list:
if t in tuples_to_search:
return t
def intersect_search_tuples_in_tuples_list(tuples_to_search, big_tuple_list):
big_tuple_set = set(big_tuple_list)
return tuples_to_search.intersection(big_tuple_set)
def get_big_data(big_tuple_list_size):
big_tuple_list = [(
data_values[randint(random_lower_limit, random_upper_limit)],
data_values[randint(random_lower_limit, random_upper_limit)],
data_values[randint(random_lower_limit, random_upper_limit)]
) for _ in range(big_tuple_list_size)]
return big_tuple_list
def get_small_data(big_tuple_list, tuples_to_search_size):
return sample(big_tuple_list, tuples_to_search_size)
def get_required_data(big_tuple_list_size, tuples_to_search_size):
big_tuple_list = get_big_data(big_tuple_list_size)
tuples_to_search = get_small_data(big_tuple_list, tuples_to_search_size)
return (big_tuple_list, tuples_to_search)
if __name__ == '__main__':
test_combinations = [(10000, 5), (20000, 5), (100000, 5)]
functions_to_test = [
normal_search_multiple_tuples_in_tuples_list,
intersect_search_tuples_in_tuples_list
]
for big_tuple_list_size, tuples_to_search_size in test_combinations:
tuples_to_search, big_tuple_list = get_required_data(big_tuple_list_size, tuples_to_search_size)
print(f'For a run of searching {tuples_to_search_size} needles in {big_tuple_list_size} size haystack')
s = set(tuples_to_search)
for func in functions_to_test:
print(f'''Time taken by {func.__name__}: {timeit.timeit('func(s, big_tuple_list)', number=1000000, globals=globals())}''')
print()
打印：
For a run of searching 5 needles in 10000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 0.16306289999999998
Time taken by intersect_search_tuples_in_tuples_list: 0.6508408
For a run of searching 5 needles in 20000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 0.16933260000000006
Time taken by intersect_search_tuples_in_tuples_list: 0.6473307000000001
For a run of searching 5 needles in 100000 size haystack
Time taken by normal_search_multiple_tuples_in_tuples_list: 0.1777871999999996
Time taken by intersect_search_tuples_in_tuples_list: 0.7130949000000002
您可以清楚地看到您不应该将列表转换为集合。最后，如果这不令人信服，这是我的基准测试，我在其中对我们尝试在搜索函数中匹配的列表元素进行集合转换，并且我们正在搜索的列表中与这些元素之一匹配的元素是最后一个元素.这应该给使用集合交集的方法带来最大的优势：
import timeit
def s1(s, lst):
s = set(s)
for i in lst:
if i in s:
return i
def s2(s, lst):
s = set(s)
s2 = set(lst)
return s & s2
lst = [(i,i) for i in range(20000)]
s = [(19999, 19999), (21000, 21000)] # match will be last element of lst
# look for a s in lst:
print(timeit.timeit('s1(s, lst)', number=1000, globals=globals()))
print(timeit.timeit('s2(s, lst)', number=1000, globals=globals()))
打印：
0.887862600000517
1.8508867000000464