如何仅为scrapy spider中的特定请求更改标头？

Question

我正在尝试使用 scrapy 构建网络爬虫。我想为蜘蛛中的单个请求更改用户代理。我尝试了下面的代码，但是在抓取过程中用户代理没有被更新。

    def start_requests(self):
    request = Request(
        "url",
        callback=self.parse_search,
        meta={'xpaths': self.xpaths},
        headers={
                "User-Agent": "Googlebot-Image/1.0"
            }

    )
    return [request]

Answer 1

您的代码运行良好（请参阅我的代码）。但是您身边的一些middleware 可能会影响您的User-Agent 标头：

class UserAgentSpider(scrapy.Spider):

    name = 'useragent_spider'
    user_agents = [
        {'title': 'Galaxy S9', 'value': 'Mozilla/5.0 (Linux; Android 8.0.0; SM-G960F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36'},
        {'title': 'iPhone', 'value': 'Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/69.0.3497.105 Mobile/15E148 Safari/605.1'},
        {'title': 'Edge', 'value': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246'},
    ]

    def start_requests(self):
        for user_agent in self.user_agents:
            yield scrapy.Request(
                url="https://www.myip.com/",
                headers={
                    'user-agent': user_agent['value'],
                },
                cb_kwargs={
                    'user_agent': user_agent['title']
                },
                callback=self.parse,
                dont_filter=True,
            )
    
    def parse(self, response, user_agent):
        with open(f"Samples/{user_agent}.htm", 'wb') as f:
            f.write(response.body)