同时运行此循环的最佳方法？

Question

我有以下代码：

data = [2,5,3,16,2,5]        

def f(x):       
    return 2*x

f_total = 0
for x in data:
    f_total += f(x)

print(f_total/len(data))

我想加快 for 循环。 （实际上代码更复杂，我想在具有许多处理核心的超级计算机中运行它）。我读过我可以使用multiprocessing 库来做到这一点，在那里我可以让python3同时运行不同的循环块，但我有点迷失了。

您能解释一下如何使用我的程序的这个最小版本吗？

谢谢！

Answer 1

import multiprocessing
from numpy import random

"""
This mentions the number of worker threads that you want to run in parallel.
Depending on the number of cores in your system you should choose the appropriate
number of threads. When you call 'map' function it will distribute the input
values in that many parts
"""
NUM_CORES = 6
data = random.rand(100, 1)

"""
+2 so that the cores are not left idle in case a thread is waiting for I/O. 
Choose by performing an empirical analysis depending on the function you are trying to compute.
It could match up to NUM_CORES as well. You can vary the chunksize as well depending on the size of 'data' that you have. 
"""
NUM_THREADS = NUM_CORES+2
CHUNKSIZE = int(len(data)/(NUM_THREADS))    


def f(x):       
    return 2*x

# This takes care of creating pool of worker threads which will be assigned the jobs
pool = multiprocessing.Pool(NUM_THREADS)

# map vs imap. If the data is large go for imap else map is also good.
it = pool.imap(f, data, chunksize=CHUNKSIZE)

f_total = 0
# Iterate and sum up the result
for value in it:
    f_total += sum(value)

print(f_total/len(data))

Why choose imap over map?