为什么scrapy 说我正在迭代一个“itemMeta”对象？

Question

我正在尝试抓取一个网站，以进一步了解 scrapy 的工作原理。我对包请求和 bs4 (BeautifulSoup) 有一点经验。我在我的 Ubuntu 20.04.1 LTS 机器上的 miniconda3 环境中工作。我使用 python 3.7。

我创建了一个名为“PostscrapeItem”的项目，它只有一个属性：full_text = scrapy.Field()。已经被scrapy自动创建的项目结构我没有接触过。

我制作了一个蜘蛛，它只应该在这个网页上找到一个 html 标记 ('em')：https://blog.scrapinghub.com/page/1/

这是我的蜘蛛的代码：

import scrapy
from bs4 import BeautifulSoup
from postscrape.items import PostscrapeItem


class PostSpider(scrapy.Spider):
    name = "posts"

    start_urls = [
        'https://blog.scrapinghub.com/page/1/'
    ]

    def parse(self, response):
        so = BeautifulSoup(response.text, 'html.parser')
        item = PostscrapeItem()

        if so.find('em'):
            concatenated = ""
            text_samples = so.find_all('em')

            for t_s in text_samples:
                concatenated += t_s.text

            item['full_text'] = concatenated

        return PostscrapeItem

我遇到的问题是，当我在终端中使用“scrapy crawl posts”运行此代码时出现错误，它显示：“TypeError: 'ItemMeta' object is not iterable '。据我所知，我的程序中唯一存在的 ItemMeta 是对象PostscrapeItem。对我来说，我没有在我的代码中迭代这个对象。这就是我问你的原因。

这是完整的错误信息：

Traceback (most recent call last):  

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/utils/defer.py", 

line 117, in iter_errback

yield next(it)

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__

return next(self.data)

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 345, in __next__

return next(self.data)

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable

for r in iterable:

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output

for x in result:

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable

for r in iterable:

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 338, in <genexpr>

return (_set_referer(r) for r in result or ())

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable

for r in iterable:

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>

return (r for r in result or () if _filter(r))

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable

for r in iterable:

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>

return (r for r in result or () if _filter(r))

File "/home/luc/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 64, in _evaluate_iterable

for r in iterable:

TypeError: 'ItemMeta' object is not iterable`

提前感谢您，让我知道如何提高我的问题的清晰度和质量。

卢克

Answer 1

您返回的不是项目，而是项目类对象。
Scrapy 在从蜘蛛返回时尝试对其进行迭代，因此您将获得 TypeError。

只需将最后一行更正为 return item 即可修复您的代码。

附带说明一下，scrapy 有自己的解析实用程序，因此无需导入和使用 BS。

Answer 2

根据@stranac 的回答，我已更正完整代码及其工作。

import scrapy
from bs4 import BeautifulSoup

class PostscrapeItem(scrapy.Item):
    full_text = scrapy.Field()


class PostSpider(scrapy.Spider):
    name = "posts"

    start_urls = [
        'https://blog.scrapinghub.com/page/1/'
    ]

    def parse(self, response):
        so = BeautifulSoup(response.text, 'html.parser')
        item = PostscrapeItem()

        if so.find('em'):
            concatenated = ""
            text_samples = so.find_all('em')

            for t_s in text_samples:
                concatenated += t_s.text

            item['full_text'] = concatenated

        return item