在scrapy中获取回调的返回值-python框架

Question

我正在寻找一种从网站的所有 URL 获取电子邮件的方法 - 基本上，index.php、contact.php 和其他 URL。我的爬虫遍历每个页面，发出请求并从每个响应源代码中获取电子邮件。如果 index.php 中有一封电子邮件，我想存储它并添加更多可以从 contact.php 中找到的电子邮件。这意味着item['emails'] 将保存一个电子邮件列表，该列表将随着更多页面被抓取而扩展。

我的问题是因为我使用extract_emails() 作为回调来获取电子邮件（在get_emails() 内）我如何让它返回电子邮件并将其附加到稍后将分配给item['emails'] 的列表中

def parse_site(self, response):
            item = response.meta['item']
            item['websites'] = response.xpath("//div[@class='company-contact-information']/table/tr/td/a/@href").extract()

            item['websites'] = self.remove_blacklist_links(item['websites'])
            # item['websites'] = ['http://www.kai-hsiang.com.tw','http://www.yangshitex.com']
            print(item['websites'])

            time.sleep(1)

            for web in item['websites']:
                request = scrapy.Request(web, callback=self.get_emails, dont_filter=True)
                print(web)
                print(request)
                time.sleep(2)
                yield request

        def get_emails(self, response):
            # the_leads = []
            item = response.meta['item']
            for l in response.xpath('//a'):
                link = l.xpath('@href').extract()[0]
                print(link)
                ju = urlparse.urljoin(response.url,  link)
                # the_leads.append(self.extract_email) --> Just added this now to explain what is 
                # possible if not for the request and yield
                request = scrapy.Request(ju, callback=self.extract_emails, dont_filter=True, meta={'item': item}) # ---> How do i get a return value for extract_emails ?
                print(response.url, ju, link, "*" * 100)
                yield request
            # item['emails'] = all_leads
            # return item


        def extract_emails(self, response):
            item = response.meta['item']

            regex = re.compile(r'([\w\-\.]{1,100}@(\w[\w\-]+\.)+[\w\-]+)')
            emails = list(set(regex.findall(response.body)))
            all_emails = [email[0].lower() for email in emails]
            item['emails'] = all_emails
            # print(item)
            # print("*" * 20)

            return item  # this overrites the emails

Answer 1

将新列表附加到现有列表中：

item['emails'] = item['emails'] + all_emails