Scrapy 不会更改代理

Question

在尝试使用 Scrapy 测试代理时，我遇到了一个问题。我想用 httpbin.org 检查代理，并制作爬虫：

class CheckerSpider(scrapy.Spider):
    name = "checker"
    start_urls = (
        'https://www.httpbin.org/ip'
    )
    connection = get_connection()

    def start_requests(self):

        with self.connection.cursor() as cursor:
            limit = int((datetime.now() - datetime(1970, 1, 1)).total_seconds()) - 3600
            q = """ SELECT *
                    FROM {}
                    WHERE active = 1 AND last_checked <= {} OR last_checked IS NULL;""".format(DB_TABLE, limit)
            cursor.execute(q)
            proxy_list = cursor.fetchall()

        for proxy in proxy_list[:15]:
            word = get_random_word()
            req = scrapy.Request(self.start_urls, self.check_proxy, dont_filter=True)
            req.meta['proxy'] = 'https://{}:8080'.format(proxy['ip'])
            req.meta['item'] = proxy
            user_pass = base64.encodestring('{}:{}'.format(PROXY_USER, PROXY_PASSWORD))
            req.headers['Proxy-Authorization'] = 'Basic {}'.format(user_pass)
            req.headers['User-Agent'] = get_user_agent()
            yield req

    def check_proxy(self, response):
        print response.request.meta['proxy']
        print response.meta['item']['ip']
        print response.body

但是当我测试它时，我发现 Scrapy 仅使用 5 个代理连接到 url，然后没有更改它。示例输出（只是弄乱了 IP）：

2016-02-23 14:54:36 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.130:8080
192.168.100.130
{
  "origin": "192.168.100.130"
}

2016-02-23 14:54:36 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.131:8080
192.168.100.131
{
  "origin": "192.168.100.131"
}
2016-02-23 14:54:37 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.132:8080
192.168.100.132
{
  "origin": "192.168.100.132"
}

# Here Scrapy used wrong proxy to connect to site.
2016-02-23 14:54:37 [scrapy] DEBUG: Crawled (200) <GET https://www.httpbin.org/ip> (referer: None)
https://192.168.100.134:8080
192.168.100.134
{
  "origin": "192.168.100.130"
}

可能是我做错了什么？任何的想法？谢谢。

更新： 实际上，现在我正在使用中间件来为请求添加代理。我把它按顺序放在中间件中：

DOWNLOADER_MIDDLEWARES = {
    'checker.middlewares.ProxyCheckMiddleware': 100,
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}

但我有同样的结果。这是我添加代理的自定义中间件：

class ProxyCheckMiddleware(object):

    def process_request(self, request, spider):
        if 'proxy' not in request.meta:
            request.meta['proxy'] = 'https://{}:8080'.format(request.meta['item']['ip'])
            request.meta['handle_httpstatus_list'] = [302, 503]
            user_pass = base64.encodestring('{}:{}'.format(PROXY_USER, PROXY_PASSWORD))
            request.headers['Proxy-Authorization'] = 'Basic {}'.format(user_pass)

UPD。 到目前为止，这似乎是 Scrapy 中的一个错误。看这里的对话：https://github.com/scrapy/scrapy/issues/1807

Answer 1

在故事的最后，这是 Scrapy 中的一个错误，该错误已在 1.1.0 版中修复（请参阅conversation）。非常感谢 redapple 和 rverbitsky 的帮助！

Answer 2

你试过this吗？

按照设置说明，拥有一个包含指定格式的代理列表的文本文件，并通过它运行请求。它将使用的代理随机化，并丢弃在设定的尝试次数后失败的代理。可以强烈推荐它，目前将它与 hidemyass.com 的代理列表一起使用

Answer 3

在您的 middlewares.py 文件中试用此 ProxyMiddleware。

class ProxyMiddleware(object):

    def process_request(self, request, spider):
        request.meta['proxy'] == 'https://{}:8080'.format(request.meta.get('item').get('ip'))

        # If the proxy needs auth (you will also need to import base64 
        # proxy_auth = "username:password"
        # encoded_auth = base64.encodestring(proxy_auth)

        # request.headers['Proxy-Authorization'] = 'Basic ' + encoded_auth
        return request

在您的settings.py 文件中：

DOWNLOADER_MIDDLEWARES = {
    'checker.middlewares.ProxyMiddleware': 100,
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110
}