使用 Queue() 进行多处理：TypeError: can't pickle _thread.lock objects

Question

这是我的代码：

def search(page, sort, start, end, q):
    print(mp.current_process())
    results = req.request_buff_market_page(page, sort, start, end)

    for i in results:
        if(i[1] >= 20 and 'Souvenir' not in i[0]):
            q.put(i)


if __name__ == '__main__':
    page = 1
    q = queue.Queue()
    with mp.Pool(processes=mp.cpu_count()) as pool:
        print(page)
        pool.starmap(search, (page, 'asc', 200, 205, q))
        page += 1

    while not(q.empty()):
        print(q.get())

输出：

 Traceback (most recent call last):
      File "C:\Users\Andrew\Desktop\ultralight\mp_buff.py", line 33, in <module>
        pool.starmap(search, (page, 'asc', 200, 205, q))
    .........
    TypeError: can't pickle _thread.lock objects

我不确定是什么导致了这个错误。错误发生在这一行：

pool.starmap(search, (page, 'asc', 200, 205, q))

编辑：将我的代码更改为使用 multiprocessing.Process()

 for i in range(mp.cpu_count()):
     workers.append(mp.Process(target=search, args=(page, 'asc', 200, 230, q_search)))
     page += 1
 for i in workers: i.start()
 for i in workers: i.join()

Answer 1

您将多处理池视为多线程池，这不是一回事。查看multiprocessing.pool.ThreadPool。