为python初学者处理异步pycurl请求答案

【问题标题】：Async pycurl requests processing for a python beginner为python初学者处理异步pycurl请求
【发布时间】：2014-11-30 22:40:35
【问题描述】：

我正在尝试结合 program A 的异步功能

program B启用的超简单的基于字符串的逻辑

#pseudocode 
    label beginning
    sleep(10)
    if substring in someString:
        print "It's not happening!!!"
        goto beginning

片段 2：

 #unique verification variable automatically gets generated every request 
 c.setopt(pycurl.HTTPHEADER, ['verification: ' + verification ])

基本上，如果第一次请求响应 html 没有返回特定字符串。必须在 10 秒后发送具有相同验证码的请求。这一切都必须以不接触硬盘（仅内存）的方式异步发生，因此它可以以每秒 1k> 个请求的速度执行。

Python 以某种纯洁崇拜的名义缺少 goto，这让我在解决这个问题时感到头疼。

重心似乎围绕着这些函数：c.setopt(pycurl.WRITEDATA,) vs c.setopt(pycurl.WRITEFUNCTION,) m = pycurl.CurlMulti() m.handles.append(c)

欢迎任何关于如何最好地解决这个难题的建议。我正在寻找的主要可能是伪代码/逻辑的一般性+我应该研究的一些功能建议，一旦我有了一般的蓝图，我应该能够自己拼凑起来。

【问题讨论】：

标签： python curl asynchronous pycurl

【解决方案1】：

from StringIO import StringIO

import pycurl

class CurlStream(object):
    """"""
    curl_count = 0
    curl_storage = []

    def __init__(self):
        self.curl_multi = pycurl.CurlMulti()

    def add_request(self, request, post_fields=None):
        self.curl_count += 1
        curl = self._create_curl(request, post_fields)
        self.curl_multi.add_handle(curl)

    def perform(self):
        while self.curl_count:
            while True:
                response, self.curl_count = self.curl_multi.perform()
                if response != pycurl.E_CALL_MULTI_PERFORM:
                    break
            self.curl_multi.select(1.0)

    def read_all(self):
        for response in self.curl_storage:
            print response.getvalue() # this does nothing --prints blank lines

    def close(self):
        self.curl_multi.close()

    def _create_curl(self, request, post_fields):
        curl = pycurl.Curl()
        curl.setopt(curl.URL, request)
        curl.setopt(curl.WRITEFUNCTION, self.write_out) # now passing own method
        curl.setopt(curl.TIMEOUT, 20)
        # Below is the important bit, I am now adding each curl object to a list
        self.curl_storage.append(curl)
        return curl

    def write_out(self, data):
        print 'Data len', len(data)
        print data
        return len(data)


def main():
    curl_stream = CurlStream()
    curl_stream.add_request('http://www.google.com')
    curl_stream.add_request('http://www.tomdickin.com')
    curl_stream.perform()
    curl_stream.read_all()
    curl_stream.close()

if __name__ == '__main__':
    main()

How can I get the response body from pycurl multi curl requests

该答案的代码看起来不错，并且只有在我按照它应该做的事情后最后运行它时才有效

Traceback (most recent call last):
  File "Untitled 2.py", line 55, in <module>
    main()
  File "Untitled 2.py", line 53, in main
    curl_stream.read_all()
  File "Untitled 2.py", line 28, in read_all
    print response.getvalue() # this does nothing --prints blank lines
AttributeError: getvalue

【讨论】：