Scrapy 抓取数据但没有输出到文件

Question

尽管成功地能够在 scrapy shell 中执行大部分行，但我一直得到空白的 json 文件。 当我使用我的课程机器人运行命令 scrapy crawl courses 时：

from scrapy.spiders import CrawlSpider
from scrapy.linkextractors import LinkExtractor
from tutorial.items import CoursesItem
from bs4 import BeautifulSoup
import scrapy

class CoursesSpider(CrawlSpider):
    name = 'courses'
    allowed_domains = ['guide.berkeley.edu']
    start_urls = ['http://guide.berkeley.edu/courses/ast', 
                   ]

def parse(self, response):
    soup = BeautifulSoup(response.body_as_unicode(), 'lxml')
    items = []
    for course_info, course_desc, course_req in zip(soup.find_all('p',class_='courseblocktitle'), \
                                                soup.find_all('p', class_='courseblockdesc'), \
                                                soup.find_all('div', class_='course-section')):     
        item = CoursesItem()
        item['title'] = course_info.text
        item['description'] = course_desc.text
        item['requirements'] = course_req.text
        yield items

并且通过 settings.py 存在

BOT_NAME = 'courses'

SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0.3'

# ITEM_PIPELINES = {
#   'tutorial.pipelines.JsonExportPipeline': 300
# }

FEED_URI = 'output.json'
FEED_FORMAT = 'json'

正如您在评论部分看到的那样，我也尝试过制作管道。 我的管道文件如下所示：

from scrapy import signals
from scrapy.exporters import JsonLinesItemExporter

class JsonExportPipeline(object):

    def __init__(self):
        self.files = {}

    @classmethod
    def from_crawler(cls, crawler):
        pipeline = cls()
        crawler.signals.connect(pipeline.spider_opened, signals.spider_opened)
        crawler.signals.connect(pipeline.spider_closed, signals.spider_closed)
        return pipeline

    def spider_opened(self, spider):
        file = open('%s_spider.json' % spider.name, 'w+b')
        self.files[spider] = file
        self.exporter = JsonLinesItemExporter(file)
        self.exporter.start_exporting()

    def spider_closed(self, spider):
        self.exporter.finish_exporting()
        file = self.files.pop(spider)
        file.close()

    def process_item(self, item, spider):
        self.exporter.export_item(item)
        return item

但我觉得这可能不是错误所在，尽管这是可能的，因为我主要遵循了我找到的几个教程。

我使用 BeautifulSoup 来简化我选择项目的方式。

最后但并非最不重要的是，终端在我运行后看起来是这样的。

2015-08-07 23:58:44 [scrapy] INFO: Scrapy 1.0.1 started (bot: courses)
2015-08-07 23:58:44 [scrapy] INFO: Optional features available: ssl, http11
2015-08-07 23:58:44 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tu
torial.spiders', 'FEED_URI': 'output.json', 'SPIDER_MODULES': ['tutorial.spiders
'], 'BOT_NAME': 'courses', 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv
:39.0) Gecko/20100101 Firefox/39.0.3', 'FEED_FORMAT': 'json'}
2015-08-07 23:58:44 [scrapy] INFO: Enabled extensions: CloseSpider, FeedExporter
, TelnetConsole, LogStats, CoreStats, SpiderState
2015-08-07 23:58:44 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddl
eware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultH
eadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMidd
leware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-08-07 23:58:44 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-08-07 23:58:44 [scrapy] INFO: Enabled item pipelines:
2015-08-07 23:58:44 [scrapy] INFO: Spider opened
2015-08-07 23:58:44 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 i
tems (at 0 items/min)
2015-08-07 23:58:44 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2015-08-07 23:58:45 [scrapy] DEBUG: Redirecting (301) to <GET http://guide.berke
ley.edu/courses/ast/> from <GET http://guide.berkeley.edu/courses/ast>
2015-08-07 23:58:45 [scrapy] DEBUG: Crawled (200) <GET http://guide.berkeley.edu
/courses/ast/> (referer: None)
2015-08-07 23:58:45 [scrapy] INFO: Closing spider (finished)
2015-08-07 23:58:45 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 537,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 22109,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/301': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 8, 8, 6, 58, 45, 600000),
 'log_count/DEBUG': 3,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2015, 8, 8, 6, 58, 44, 663000)}
2015-08-07 23:58:45 [scrapy] INFO: Spider closed (finished)

我已经彻底地运行了大部分选项。运行 --parse 的单数选项告诉我，我正在解析项目，但即便如此，我想知道在哪里可以超越解析错误修复（即输出到 json）。最终，我想将所有这些数据通过管道传输到数据库中。

我知道有很多内容需要查看，但感谢您提供任何帮助，谢谢！

Answer 1

你写错字了。在解析函数中更改项目 -> 项目。

def parse(self, response):
    soup = BeautifulSoup(response.body_as_unicode(), 'lxml')
    items = []
    for ...
        item = CoursesItem()
        item['title'] = course_info.text
        item['description'] = course_desc.text
        item['requirements'] = course_req.text
        yield items # -> item

`