将属性分配给 Scrapy 中的错误项目答案

【问题标题】：assign properties to wrong items in Scrapy将属性分配给 Scrapy 中的错误项目
【发布时间】：2018-05-24 09:21:41
【问题描述】：

代码

蜘蛛.py

...
def parse(self, response):
    for one_item in response.xpath('path1'):
        item = ProjectItem()
        request = scrapy.Request(one_item.xpath('path2'), callback=self.parse2)
        request.meta['item'] = item
        yield request

property = []
def parse2(self, response)
   item = response.meta['item']

   for x in response.xpath('path3')
       self.property.append('path4')

   next_page = response.xpath('path5')
   if next_page is not None:
       request2 = scrapy.Request(next_page, callback=self.parse2)
       request2.meta['item'] = item
       yield request2
   else:
       item['field'] = self.property
       self.property = []
       yield item

问题是当蜘蛛爬到next_page时。一些self.property 分配给了错误的项目。我不知道如何修复它。

【问题讨论】：

标签： python web-scraping scrapy scrapy-spider

【解决方案1】：

self.property 是一个类属性，在所有对 parse2 的调用之间共享，您无法控制对 parse2 的每次调用的顺序。

要解决您需要在元内或作为项目属性传递属性列表：

def parse(self, response):
    for one_item in response.xpath('path1'):
        item = ProjectItem()
        item['field'] = []
        request = scrapy.Request(one_item.xpath('path2'), callback=self.parse2)
        request.meta['item'] = item
        yield request

def parse2(self, response)
   item = response.meta['item']
   for x in response.xpath('path3')
       item['field'].append('path4')

   next_page = response.xpath('path5')
   if next_page is not None:
       request2 = scrapy.Request(next_page, callback=self.parse2)
       request2.meta['item'] = item
       yield request2
   else:
       yield item

【讨论】：