scrapy - 在 http 代码上调用 process_exception答案

【问题标题】：scrapy - call process_exception on http codescrapy - 在 http 代码上调用 process_exception
【发布时间】：2015-03-26 10:32:53
【问题描述】：

我想更改代理服务取决于 http 响应代码（例如在 code=500 或 404 上）我想触发 process_exception 更改代理地址。我创建了自己的proxyMiddleware，并在process_request 中设置了代理，例如。当发生超时代理时，默认调用process_exception。但是如何在自定义 http 状态下触发它？

来自 scrapy 文档：

当下载处理程序或 process_request()（来自下载器中间件）引发异常（包括 IgnoreRequest 异常）

但我不知道如何实现。

编辑我的蜘蛛代码

class Spider1(CrawlSpider):
#     pageNumber = 0

    keyword = ''
    page = range(0, 40, 10)

    allowed_domains = ['http://somedomain.com/search.html?query=football']
    start_urls = ['http://somedomain.com/search.html?query=football']
    rules = (Rule (LxmlLinkExtractor(), callback="parse", follow=True),) 

    def parse(self, response):
        return item

我的设置.py：

DOWNLOADER_MIDDLEWARES = {
    't.useragentmiddleware.RandomUserAgentMiddleware': 400,
    'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
    'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
    'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware':720,
    't.cookiesmiddleware.CookiesMiddleware': 700,
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 760,
    't.proxymiddleware.ProxyMiddleware': 750
}
REDIRECT_ENABLED = True

和proxymiddleware.py：

import json, os, random, socket
import t as spider1
import scrapy.exceptions as exception

socket.setdefaulttimeout(5)

class ProxyMiddleware(object):

    proxy = ''
    proxyList = []
    handle_httpstatus_list = [302, 400]

    def __init__(self, settings):
        f = open(t.location + '/data/proxy.json')
        self.proxyList = json.load(f)['proxy']
        f.close()

    def process_request(self, request, spider):
        if 'proxy' in request.meta:
            return

        self.proxy = 'http://' + random.choice(self.proxyList)

        os.environ['http_proxy'] = self.proxy
        request.meta['proxy'] = self.proxy


    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def process_exception(self, request, exception, spider):
        proxy = request.meta['proxy']

        try:
            del self.proxyList[self.proxyList.index(proxy[8:])]
        except ValueError:
            pass
        prox = 'http://' + random.choice(self.proxyList)
        request.meta['proxy'] = prox
        os.environ['http_proxy'] = prox

    def process_response(self, request, response, spider):
        '''this doesn't work'''
        #raise exception.NotConfigured()

【问题讨论】：

标签： python proxy scrapy

【解决方案1】：

有效的 HTTP 状态代码不是“异常”，因此它们通过 process_response 路由。提取一个方法并从process_exception 和process_response 调用它。

CHANGE_PROXY_STATUS_LIST = [502, 404]

class ProxyMiddleware(object):
    def change_proxy(request):
        # Change proxy here
        # Then check number of retries on the request 
        # and decide if you want to give it another chance.
        # If not - return None else
        return request  

    def process_exception(self, request, exception, spider):
        return_request = change_proxy(request)
        if return_request: 
            return return_request

    def process_response(self, request, response, spider):
        if response.status in CHANGE_PROXY_STATUS_LIST:
            return_request = change_proxy(request)
            if return_request: 
                return return_request
        return response

【讨论】：