解析回调时的 Scrapy 请求“IndentationError：意外缩进”答案

【问题标题】：Scrapy Request 'IndentationError: unexpected indent' on parse callback解析回调时的 Scrapy 请求“IndentationError：意外缩进”
【发布时间】：2020-11-10 11:38:48
【问题描述】：

我正在使用 Scrapy CLI 并在 ubuntu 18 服务器上运行它，我试图避免在 start_urls 属性中硬编码一堆 url，而是在我的底部调用“yield scrapy.Request()”解析。我正在抓取的网站相当基本，并且在 2014-2030 年间有不同的页面。在我的代码底部，我有一个 if() 函数来检查当前年份并将刮板移动到下一年的页面。一般来说，我是scrapy的新手，所以我不确定我是否正确调用了scrapy.Request()方法。这是我的代码：

import scrapy

from .. import items

class EventSpider(scrapy.Spider):
    name = "event_spider"
    start_urls = [
        "http://www.seasky.org/astronomy/astronomy-calendar-2014.html",
    ]
    user_agent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"
    start_year = 2014
    
    #response is the website
    def parse(self, response):
        CONTENT_SELECTOR = 'div#right-column-content ul li'
        
        for astro_event in response.css(CONTENT_SELECTOR):
            NAME_SELECTOR = "p span.title-text ::text"
            DATE_SELECTOR = "p span.date-text ::text"
            DESCRIPTION_SELECTOR = "p ::text"
            
            item = items.AstroEventsItem()
            
            item["title"] = astro_event.css(NAME_SELECTOR).extract_first()
            item["date"] = astro_event.css(DATE_SELECTOR).extract_first()
            item["description"] = astro_event.css(DESCRIPTION_SELECTOR)[-1].extract()
            
            yield item
               
        #Next page code:
        #Goes through years 2014 to 2030
        if(self.start_year < 2030):
            self.start_year = self.start_year + 1
            new_url = "http://www.seasky.org/astronomy/astronomy-calendar-" + str(self.start_year) + ".html"
            print(new_url)
            yield scrapy.Request(new_url, callback = self.parse)

这是我成功抓取第一页后收到的错误：

2020-11-10 05:25:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.seasky.org/astronomy/astronomy-calendar-2015.html> (referer: http://www.seasky.org/astronomy/astronomy-calendar-2014.html)
2020-11-10 05:25:50 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.seasky.org/astronomy/astronomy-calendar-2015.html> (referer: http://www.seasky.org/astronomy/astronomy-calendar-2014.html)
Traceback (most recent call last):
  File "/home/jcmq6b/.local/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
StopIteration: <200 http://www.seasky.org/astronomy/astronomy-calendar-2015.html>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/defer.py", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/spidermw.py", line 58, in process_spider_input
    return scrape_func(response, request, spider)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/core/scraper.py", line 149, in call_spider
    warn_on_generator_with_return_value(spider, callback)
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/misc.py", line 245, in warn_on_generator_with_return_value
    if is_generator_with_return_value(callable):
  File "/usr/local/lib/python3.6/dist-packages/scrapy/utils/misc.py", line 230, in is_generator_with_return_value
    tree = ast.parse(dedent(inspect.getsource(callable)))
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    def parse(self, response):
    ^
IndentationError: unexpected indent

我想我可能没有传递正确的参数来回调 parse 方法，但我不确定。任何帮助深表感谢！如果我需要发布更多信息，请告诉我。

【问题讨论】：

您的错误提示 unexpected indent 您可能在 def 之前有错误数量的空格，在任何情况下，您在此处发布的代码都不会出现缩进错误。我试过了，它可以编译
奇怪的是，如果我取出代码的底部部分 if() 语句，那么它运行良好，没有缩进错误。无论如何，我会仔细检查我的间距，看看它是否运行
我认为缩进错误是scrapy中的一个错误。这是 GitHub 上的一个相关问题：github.com/scrapy/scrapy/issues/4477 但没有专门针对此问题的问题。

标签： python python-3.x web-scraping scrapy

【解决方案1】：

对于任何偶然发现此问题的人，我没有找到缩进错误的原因，但我确实找到了解决方法，方法是将我的代码分成两种不同的解析方法：

import scrapy

from .. import items

class EventSpider(scrapy.Spider):
    name = "event_spider"
    start_urls = ["http://www.seasky.org/astronomy/astronomy-calendar-2014.html"]
    user_agent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"
    start_year = 2014
    
    #Next page code:
    def parse(self, response):
        #Goes through years 2014 to 2030 from the href links at top of page
        for href in response.css("div#top-links div h3 a::attr(href)"):
            url = response.urljoin(href.extract())
            yield scrapy.Request(url, callback=self.parse_contents)

            
    
    #parses items for database
    def parse_contents(self, response):
        
        CONTENT_SELECTOR = 'div#right-column-content ul li'
        
        for astro_event in response.css(CONTENT_SELECTOR):
            NAME_SELECTOR = "p span.title-text ::text"
            DATE_SELECTOR = "p span.date-text ::text"
            DESCRIPTION_SELECTOR = "p ::text"
            
            item = items.AstroEventsItem()
            
            item["title"] = astro_event.css(NAME_SELECTOR).extract_first()
            item["date"] = astro_event.css(DATE_SELECTOR).extract_first()
            item["description"] = astro_event.css(DESCRIPTION_SELECTOR)[-1].extract()
            
            yield item

第一个解析是从网站上列出的 href 中获取 url。然后它为每个 href 调用第二个解析方法 parse_contents 并将从页面中抓取的信息处理为 MongoDB 的项目。如果有人遇到类似问题，希望这可以帮助他们。

【讨论】：