在scrapy spider代码中替代scrapy.contrib

Question

我在一本关于网络抓取的教科书中找到了这个示例代码。运行蜘蛛后它显示错误并发现 scrapy.contrib 在 1.16 版本的 scrapy 中被删除。我应该如何改变它才能工作。顺便说一句，我是网络抓取的新手。

from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule

class ArticleSpider(CrawlSpider):
    name = 'articles'
    allowed_domains = ['wikipedia.org']
    start_urls = ['https://en.wikipedia.org/wiki/'
                  'Benevolent_dictator_for_life']
    rules = [Rule(LinkExtractor(allow='.*'), callback='parse_items',
                  follow=True)]

    def parse_items(self, response):
        url = response.url
        title = response.css('h1::text').extract_first()
        text = response.xpath('//div[@id="mw-content-text"]//text()').extract()
        lastUpdated = response.css('li#footer-info-lastmod::text').extract_first()
        lastUpdated = lastUpdate.replace(
            'This page was last edited on ','')
        print('URL is: {}'.format(url))
        print('title is: {}'.format(title))
        print('text is: {}'.format(text))
        print('Last updated: {}'.format(lastUpdated))

Answer 1

在较新版本的scrapy中，您可以简单地导入以下模块

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
# add the rest of the code

阅读更多来自docs。