如何使用scrapy回调在两个蜘蛛之间传递参数

Question

我有两个scrapy，第一个抓取站点地图并提取网址并将其放入txt文件中，第二个读取并逐行抓取此网址。

我的代码如下：

class sitemapSpider(SitemapSpider):
    name = "filmnetmapSpider"
    sitemap_urls = ['http://filmnet.ir/sitemap.xml']
    sitemap_rules = [
    ('/series/', 'parse_item')
    ]
    storage_file = 'urls.txt'

    def parse_item(self, response):
         videoid = response.url

         with open(self.storage_file, 'a') as handle:
            yield handle.writelines(videoid + '\n')

第二只蜘蛛：

class filmnetSpider(scrapy.Spider):
      name = 'filmnetSpider'

    def start_requests(self):
        with open('urls.txt') as fp:
            for line in fp:
                yield Request(line.strip(), callback=self.parse_website)

    def parse_website(self, response):
        hxs = HtmlXPathSelector(response)
        url = hxs.xpath('//script[@type="application/ld+json"]/text()').extract()
        url = ast.literal_eval(json.dumps(url))
        url = url[1]
        obj = json.loads(url)
        poster = obj['image']
        name = obj['name']
        description = obj['description']

如何更改代码以删除对文件的读/写？

如何在其中使用回调？

注意：此代码不适用于一个scrapy spider；代码为：两个给定的scrapy + 波纹管代码，如doc中所说的示例

process = CrawlerProcess()
process.crawl(filmnetSpider)
process.crawl(sitemapSpider)
process.start()

Answer 1

这应该可行：

class sitemapSpider(SitemapSpider):
    name = "filmnetmapSpider"
    sitemap_urls = ['http://filmnet.ir/sitemap.xml']
    sitemap_rules = [
    ('/series/', 'parse_item')
    ]

    def parse_item(self, response):
         videoid = response.url
         yield Request(videoid, callback=self.parse_website)

    def parse_website(self, response):
        hxs = HtmlXPathSelector(response)
        url = hxs.xpath('//script[@type="application/ld+json"]/text()').extract()
        url = ast.literal_eval(json.dumps(url))
        url = url[1]
        obj = json.loads(url)
        poster = obj['image']
        name = obj['name']
        description = obj['description']