使用 Splash 在 Scrapy 中运行 CrawlerProcess

Question

我有一个scrapy+splash 文件来抓取数据。现在我想通过脚本运行我的scrapy文件，所以我使用CrawlerProcess。我的文件是这样的：

import scrapy
from scrapy_splash import SplashRequest
from scrapy.crawler import CrawlerProcess
class ProvinceSpider(scrapy.Spider):
    name = 'province'

    def start_requests(self):
        url = "https://e.vnexpress.net/covid-19/vaccine"

        yield SplashRequest(url=url,callback=self.parse)

    def parse(self, response):
        provinces = response.xpath("//div[@id='total_vaccine_province']/ul[@data-weight]")
        for province in provinces:
            yield{
                'province_name':province.xpath(".//li[1]/text()").get(),
                'province_population':province.xpath(".//li[2]/text()").get(),
                'province_expected_distribution':province.xpath(".//li[3]/text()").get(),
                'province_actual_distribution':province.xpath(".//li[4]/text()").get(),
                'province_distribution_percentage':province.xpath(".//li[5]/div/div/span/text()").get(),
            }

process = CrawlerProcess(settings={
    "FEEDS": {
        "province.json": {"format": "json"},
    },
})

process.crawl(ProvinceSpider)
process.start() # the script will block here until the crawling is finished

但是当我跑步时

python3 province.py

它没有连接到启动服务器，因此无法抓取数据。知道我做错了哪一部分吗？提前谢谢

Answer 1

原来您实际遇到的问题已被以下答案覆盖：Answer

快速分解（如果您对细节不感兴趣）：

转到 settings.py 并添加一个USER-AGENT，在我的例子中，我将其保留为：

USER_AGENT = 'testit (http://www.yourdomain.com)'

然后运行您的爬虫，它应该可以工作。为什么？你的scrapy被网站屏蔽了。

输出：

2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'HCMC', 'province_population': '7.2M', 'province_expected_distribution': '13.8M', 'province_actual_distribution': '14.6M', 'province_distribution_percentage': '100%'}
2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'Hanoi', 'province_population': '6.2M', 'province_expected_distribution': '11.4M', 'province_actual_distribution': '12.3M', 'province_distribution_percentage': '99,2%'}
2021-12-26 13:15:32 [scrapy.core.scraper] DEBUG: Scraped from <200 https://e.vnexpress.net/covid-19/vaccine>
{'province_name': 'Dong Nai', 'province_population': '2.4M', 'province_expected_distribution': '4.3M', 'province_actual_distribution': '5M', 'province_distribution_percentage': '100%'}
...
...

这是我的自定义设置：

BOT_NAME = 'testing'
SPIDER_MODULES = ['testing.spiders']
NEWSPIDER_MODULE = 'testing.spiders'
SPLASH_URL = 'http://localhost:8050'
USER_AGENT = 'testing (http://www.yourdomain.com)'
ROBOTSTXT_OBEY = False
DEFAULT_REQUEST_HEADERS = {
   'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15'
}
SPIDER_MIDDLEWARES = {
    'testing.middlewares.TestingSpiderMiddleware': 100,
}
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'