如何将 start_urls 传递给scrapy

Question

根据here的建议，我正在尝试：

scrapy crawl spider-name -a start_urls="https://start-url.com/"

我明白了：

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/scrapy/core/engine.py", line 129, in _next_request
    request = next(slot.start_requests)
  File "/usr/local/lib/python3.9/site-packages/scrapy/spiders/__init__.py", line 77, in start_requests
    yield Request(url, dont_filter=True)
  File "/usr/local/lib/python3.9/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/usr/local/lib/python3.9/site-packages/scrapy/http/request/__init__.py", line 73, in _set_url
    raise ValueError(f'Missing scheme in request url: {self._url}')

要重现，请运行以下命令：

scrapy startproject example_project
cd example_project
scrapy genspider spider1 https://stackoverflow.com
scrapy crawl spider1 -a start_urls="https://stackoverflow.com"

Answer 1

命令scrapy genspider 生成此代码：

import scrapy


class Spider1Spider(scrapy.Spider):
    name = 'spider1'
    allowed_domains = ['https://stackoverflow.com']
    start_urls = ['https://stackoverflow.com/']

    def parse(self, response):
        pass

这不会将start_urls 作为命令行参数处理。为此，请按照您链接的指南进行操作。类似的东西

import scrapy


class Spider1Spider(scrapy.Spider):
    name = 'spider1'
    allowed_domains = ['https://stackoverflow.com']

    def __init__(self, *args, **kwargs):
        super(Spider1Spider, self).__init__(*args, **kwargs)
        self.start_urls = kwargs.get('start_urls').split(',')

    def parse(self, response):
        pass

会起作用的。

注意：self.start_urls 需要一个列表，所以如果收到一个字符串，它会报错。