如何在 python 3 中创建和启动任意数量的线程

Question

我正在制作一个从网页下载图像的脚本，并且我正在尝试使其成为多线程的，因此速度要快得多。

在下载功能中，我必须设置两个参数，因为当我设置一个（队列）时，我得到了这个错误：

TypeError: downloading() takes 1 positional arguments but 21* were given

** queue has 21 links

代码：

count = 0
queue = {"some urls", , , }
done = set()
path = 'foldername'


def downloading(queue, name):
    for imgs in queue:
        if imgs not in done:
            done.add(imgs)
            urllib.request.urlretrieve(imgs, path + '/' + imgs.split('/')[-1])
            global count
            count += 1
            print(str(count) + ' ' + name)
            print('Done:  ' + imgs.split('/')[-1])


def threads(queue):
    print('Start Downloading ...')
    th1 = Thread(target=downloading, args=(queue, "Thread 1"))
    th1.start()
    th2 = Thread(target=downloading, args=(queue, "Thread 2"))
    th2.start()
    th3 = Thread(target=downloading, args=(queue, "Thread 3"))
    th3.start()
    th4 = Thread(target=downloading, args=(queue, "Thread 4"))
    th4.start()
    th5 = Thread(target=downloading, args=(queue, "Thread 5"))
    th5.start()
    th6 = Thread(target=downloading, args=(queue, "Thread 6"))
    th6.start()
    th7 = Thread(target=downloading, args=(queue, "Thread 7"))
    th7.start()
    th8 = Thread(target=downloading, args=(queue, "Thread 8"))
    th8.start()
    th9 = Thread(target=downloading, args=(queue, "Thread 9"))
    th9.start()
    th10 = Thread(target=downloading, args=(queue, "Thread 10"))
    th10.start()

Answer 1

对所需的线程数使用简单的for 循环。当然，您应该以某种方式将它们保存到最后可能join它们：

def threads(queue):
    num_of_threads = 10
    print('Start Downloading ...')
    threads = []
    for i in range(1, num_of_threads+1):
        th = Thread(target=downloading, args=(queue, "Thread {}".format(i)))
        threads.append(th)
        th.start()
    return threads

---

正如建议的那样，pool 在这个用例中会很有帮助，因为它会优化您系统的运行：

from multiprocessing.dummy import Pool as ThreadPool

def downloading(img):
    urllib.request.urlretrieve(img, path + '/' + img.split('/')[-1])
    global count
    count += 1
    print('Done:  ' + img.split('/')[-1])

def threads(queue):
    pool = ThreadPool()
    pool.map(downloading, queue)
    pool.close()
    pool.join()

请注意，通过这种方式，您应该更改downloading 函数以接收一个作为单个图像的参数。 map 函数将可迭代项（第二个参数）的每个项目发送到一个函数（第一个参数）。这也是为什么不需要设置done 的原因，因为每张图像都会被处理一次。