Python多处理一个带有循环的大列表

Question

老实说，我什至不知道给这个问题起什么名字。我正在尝试遍历大量 URL，但一次只处理 20 个 URL（20 个基于我拥有的代理数量）。但在处理 URL 时，我还需要继续遍历代理列表。因此，例如，它将从第一个 URL 和第一个代理开始，一旦它到达第 21 个 URL，它将再次使用第一个代理。这是我下面的糟糕示例，如果有人能指出我正确的方向，将不胜感激。

import pymysql.cursors
from multiprocessing import Pool
from fake_useragent import UserAgent

def worker(args):
    var_a, id, name, content, proxy, headers, connection = args
    print (var_a)
    print (id)
    print (name)
    print (content)
    print (proxy)
    print (headers)
    print (connection)
    print ('---------------------------')

if __name__ == '__main__':
    connection = pymysql.connect(
        host = 'host ',
        user = 'user',
        password = 'password',
        db = 'db',
        charset='utf8mb4',
        cursorclass=pymysql.cursors.DictCursor
    )

    ua = UserAgent()
    user_agent = ua.chrome
    headers = {'User-Agent' : user_agent}

    proxies = [
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx',
        'xxx.xxx.xxx.xxx:xxxxx'
    ]

    with connection.cursor() as cursor:
        sql = "SELECT id,name,content FROM table"
        cursor.execute(sql)
        urls = cursor.fetchall()

    var_a = 'static'

    data = ((var_a, url['id'], url['name'], url['content'], proxies[i % len(proxies)], headers, connection) for i, url in enumerate(urls))
    proc_num = 20
    p = Pool(processes=proc_num)
    results = p.imap(worker, data)
    p.close() 
    p.join()

Answer 1

您可以使用列表来存储新进程。当您达到一定数量的项目时，请为列表中的每个进程调用join。这应该可以让您控制活动进程的数量。

if __name__ == '__main__':  
    proc_num = 20
    proc_list = []
    for i, url in enumerate(urls):
        proxy = proxies[i % len(proxies)] 
        p = Process(target=worker, args=(url, proxy))
        p.start()
        proc_list.append(p)
        if i % proc_num == 0 or i == len(urls)-1: 
            for proc in proc_list: 
                proc.join()

如果您想要恒定数量的活动进程，您可以尝试Pool 模块。只需修改worker 定义即可接收一个元组。

if __name__ == '__main__': 
    data = ((url, proxies[i % len(proxies)]) for i, url in enumerate(urls))
    proc_num = 20
    p = Pool(processes=proc_num)
    results = p.imap(worker, data)
    p.close() 
    p.join()

---

为了澄清一点，worker 函数应该接收一个元组，然后将其解包。

def worker(args):
    var_a, id, name, content, proxy, headers, connection = args
    print (var_a)
    ... etc ...

Answer 2

试试下面的代码：

for i in range(len(urls)):
    url = urls[i] # Current URL
    proxy = proxies[i % len(proxies)] # Current proxy
    # ...