线程队列工作示例

Question

如何在以下代码中将打开线程的最大值限制为 20？我知道过去曾有人问过一些类似的问题，但我特别想知道如何最好地使用队列和工作示例（如果可能的话）。

    # b is a list with 10000 items
    threads = [threading.Thread(target=targetFunction, args=(ptf,anotherarg)) for ptf in b]
    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

Answer 1

执行此操作的简单方法是使用queue.Queue 进行工作，并使用for _ in range(MAXTHREADS): threading.Thread(target=f, args=(the_queue,)).start() 启动线程。然而，我发现通过子类化Thread 更容易阅读。您的里程可能会有所不同。

import threading
import queue

class Worker(threading.Thread):
    def __init__(self, q, other_arg, *args, **kwargs):
        self.q = q
        self.other_arg = other_arg
        super().__init__(*args, **kwargs)
    def run(self):
        while True:
            try:
                work = self.q.get(timeout=3)  # 3s timeout
            except queue.Empty:
                return
            # do whatever work you have to do on work
            self.q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    Worker(q, otherarg).start()
q.join()  # blocks until the queue is empty.

如果您坚持使用某个函数，我建议您将targetFunction 包装为知道如何从队列中获取的东西。

def wrapper_targetFunc(f, q, somearg):
    while True:
        try:
            work = q.get(timeout=3)  # or whatever
        except queue.Empty:
            return
        f(work, somearg)
        q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    threading.Thread(target=wrapper_targetFunc,
                     args=(targetFunction, q, otherarg)).start()
q.join()