如何使用 scrapy-playwright 为每个请求使用新的上下文？答案

【问题标题】：How to use new contex for each request with scrapy-playwright?如何使用 scrapy-playwright 为每个请求使用新的上下文？
【发布时间】：2022-08-18 20:12:23
【问题描述】：

这是我的做法，但我不确定它是否为每个新请求创建和使用新上下文：

class TestSpider(scrapy.Spider):
    name = \'test\'
    start_urls = [...]
    cnt = 0

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url,
                                 meta={\'playwright\': True,
                                       \'playwright_context\': f\'{self.cnt}\'})

    def parse(self, response):
        self.cnt += 1
        for res in response.xpath(\'//div[@id=\"contenu\"]\'):
            url = res.xpath(\'.//h2/a/@href\').get()
            yield scrapy.Request(url=url,
                                 callback=self.get_content,
                                 meta={\'playwright\': True,
                                       \'playwright_context\': f\'{self.cnt}\'})

这段代码是在做我想做的事情还是错了？

标签： python scrapy playwright-python

【解决方案1】：

self.cnt += 1 应该在发送请求之前/之后的 for 循环中，以便在发送每个请求后创建一个带有递增数字的新上下文

Class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls = [...]
    cnt = 0

    def start_requests(self):
        for url in self.start_urls: 
            self.cnt += 1   # <------ increment the count here
            yield scrapy.Request(url=url,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

    def parse(self, response):
        for res in response.xpath('//div[@id="contenu"]'):
            url = res.xpath('.//h2/a/@href').get()
            self.cnt += 1    # <------ increment the count here
            yield scrapy.Request(url=url,
                                 callback=self.get_content,
                                 meta={'playwright': True,
                                       'playwright_context': f'{self.cnt}'})

【讨论】：