Python 线程帮助 需要

Question

看到不完全适合我的案例的例子，我是雪盲的......或者他们可能这样做。因此，如果有很好的例子，我无法用大约 3 周的 Python 经验来解释它们

我有一个脚本可以查询数据库，收集可下载电影的列表，然后将它们逐一下载到您选择的目录。而且我想让它一次下载 4 或 5 个，因为这样做需要很长时间。

这是我尝试做的简化版本，myapp 是我的数据库应用程序。然而，它似乎只是按顺序运行，即使它说它正在启动两个线程

listOfIDs 是一些容器的 id，可能有也可能没有电影，然后 versionS 返回电影文件名。

import threading
import myapp_api

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, counter):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.counter = counter
    def run(self):
        print "Starting " + self.name
        for ID in listOfIDs:
            print "\nID= " + str(ID) + "\n"
            downLoad(ID)
        print "Exiting " + self.name

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)
    print "Thread Name = " + threadName


# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)


# Start new Threads
thread1.start()
thread2.start()

print "Exiting Main Thread"

好的，所以我修改了代码以接受ShadowRanger 的建议，但它仍然只是一次下载一个，我把它塞到某个地方了吗？ ...代码现在看起来像这样。

import threading
import myapp_api
from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)

with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

最后，ShadowRanger 的所有建议都是正确的，错误在于我做错了（我想我很早就迭代了 listOfIDs，并且只将最后一个传递给函数）.. . 这是最终的工作版本。

   import threading
    import myapp_api
    from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)


with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

Answer 1

我看不到线程是如何拆分工作的。看起来他们都下载了同一套东西。

如果目标是根据已知 ID 下载一堆文件，multiprocessing 有一个 .dummy 模块，它可以像 multiprocessing 一样工作，但使用线程实现，这为您提供了一个简单的线程池：

import multiprocessing.dummy as multiprocessing
from contextlib import closing

with closing(multiprocessing.Pool(4)) as pool: # Pick your favorite number of workers
    pool.map(downLoad, listOfIDs)