多线程python

Question

有没有一种方法可以使函数多线程一次从列表中获取 5 个 URL？请在下面查看我的代码。它的python 2.7

import requests, csv, time, json, threading
from lxml import html
from csv import DictWriter

All_links = ['http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.343097&longitude=-71.123046&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.398588&longitude=-71.24505&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.394319&longitude=-71.218049&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.365396&longitude=-71.23165&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.356719&longitude=-71.250479&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.385096&longitude=-71.208399&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.334146&longitude=-71.183298&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA',
'http://www.clopaydoor.com/api/v1/dealerlocator/getdealers?latitude=42.374296&longitude=-71.182371&doorType=residential&isFirstSearch=true&isHomeDepot=false&isClopayDealer=true&radius=3000&country=USA']

target = open('completedlinks.txt','ab')
def get_data(each):
    each = each.strip('\n')
    r = requests.get(each)
    source = json.loads(r.content)
    the_file = open("output.csv", "ab")
    writer = DictWriter(the_file, source[1].keys())
    writer.writeheader()
    writer.writerows(source)
    the_file.close()
    target.write(each+'\n')
    print each+"\n--------------------------"


for each in All_links:
    try:
        get_data(each)
    except:
        pass

Answer 1

查看multiprocessing package。它实现了线程池，可以实现这一点。

更新： 添加这样的东西应该可以工作

from multiprocessing import Pool

def chunks(l, n): 
""" Yield successive n-sized chunks from l. """ 
    for i in xrange(0, len(l), n): 
        yield l[i:i+n]

def threadit(threads, links):
    for part in chunks(links, threads):
        pool = Pool(threads)
        for link in part:
            pool.apply_async(getdata, args=(link,))
        pool.close()
        pool.join()

threadit(5, All_links)