如何在 Python 中对 API 查询进行多线程处理并将结果存储在 csv 文件中？答案

【问题标题】：How do I multithread an API query in Python and store the results in a csv file?如何在 Python 中对 API 查询进行多线程处理并将结果存储在 csv 文件中？
【发布时间】：2015-06-27 21:08:40
【问题描述】：

我是 Python 新手，正在编写一个简单的查询来从 USPS API 获取信息并将结果存储在 .csv 文件中，以便以后参考。我可以成功查询 API，但我想将查询扩展到大约 220 万个查询。在 for 循环中执行此操作需要数周时间，因此我将多线程视为并行运行请求的一种方式。我有两个问题：

当我执行超过 15 个左右的线程时，我会收到连接错误。该错误类似于this question，但由于我的较小查询有效，我相信它一定是来自服务器的限制。
如何保留字典的键值而不是将其更改为“0、1、2、...”？
如果我必须一次将查询限制为小批量，我是否可以保留一个主文件，在 for 循环运行时不断追加到该文件中？我知道 Python 有 this structure 用于添加到字典

下面是我的代码的一个最小示例（数据集必须很大，因为当我做大容量时会出现错误）：

from xml.etree import ElementTree as ET
from threading import Thread
import numpy as np
import pandas as pd
import requests
import csv

# API Information
usps_uname = '536UNIVE4362'
usps_pw    = '462YK79VT194'
url = 'http://production.shippingapis.com/ShippingAPITest.dll?'
req = "StandardB"

# Single API query
def delivery(origin, destination):
  query = url + 'API=' + str(req) + '&XML=%3C' + str(req) + \
  'Request%20USERID=%22' + str(usps_uname) + '%22%3E' + \
  '%3COriginZip%3E' + "%05d" % origin + '%3C/OriginZip%3E' + \
  '%3CDestinationZip%3E' + "%05d" % destination + '%3C/DestinationZip%3E' + \
  '%3C/' + str(req) + 'Request%3E'
  data = requests.get(query, auth = (usps_uname, usps_pw))
  root = ET.fromstring(data.content)
  if root[2].text == "No Data":
    DeliveryTime = 99
  else:
    DeliveryTime = int(root[2].text)
  return DeliveryTime

# Returns delivery times from all origins to specified destination
def delivery_range(origin_range, destination, thread_index, store=None):
  store[thread_index] = [0] * len(origin_range)
  for i, x in enumerate(origin_range):
    store[thread_index][i] = delivery(x, destination)
  return store

# Threading attempt
def threaded_process(nthreads, origin_range):
  store = {}
  threads = []
  for i in range(nthreads):
    ids = origin_range.values()[i]
    destination = origin_range.keys()[i]
    t = Thread(target=delivery_range, args=(ids, destination, i, store))
    threads.append(t)
  [ t.start() for t in threads ]
  [ t.join() for t in threads ]
  return store

origin_range = {
2072: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
3063: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
6095: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
7001: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
8085: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
8691: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
15205: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
17013: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
17015: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
17339: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
18031: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
18202: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
19709: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
19720: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
21224: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
23803: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
23836: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390], 
28027: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390],
29172: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390],
29303: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390],
30344: [ 2072,  3063,  6095,  7001,  8085,  8691, 15205, 17013, 17015,
       17015, 17339, 18031, 18202, 19709, 19720, 21224, 23803, 23836,
       28027, 29172, 29303, 30344, 33182, 33570, 33811, 33897, 37090,
       37127, 37310, 37416, 40165, 40218, 40511, 41048, 42718, 46075,
       46075, 46168, 46168, 46231, 47130, 53144, 66219, 67337, 75019,
       75261, 76155, 76177, 77338, 78154, 85043, 85043, 85338, 85906,
       89030, 89408, 92374, 92408, 92408, 92551, 94560, 95206, 95304,
       95363, 98004, 98032, 98327, 98390],
}

ans = threaded_process(len(origin_range), origin_range)

writer = csv.writer(open('DeliveryTimes.csv', 'wb'))
for key, value in ans.items():
   writer.writerow([key, value])

如果您第一次没有收到错误，请再次运行代码，它应该会出错。

【问题讨论】：

如果你必须多线程，那么你可能想要写入单独的 csvs 并在最后合并。

标签： python csv dictionary python-requests python-multithreading

【解决方案1】：

您可以将您的 python 脚本设置为通过参数列表将起点和终点作为输入（例如，使用 argparse 模块），然后使用 GNU Parallel (http://www.gnu.org/software/parallel/) 以所有可能的组合调用此脚本作为参数传入。

GNU Parallel 将对您的代码进行操作系统级别的并行化，因此您不必担心在 Python 中执行此操作。

【讨论】：