如何从 Python 脚本中运行 Scrapy

Question

我是 Scrapy 的新手，我正在寻找一种从 Python 脚本运行它的方法。我找到了 2 个可以解释这一点的来源：

http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/

http://snipplr.com/view/67006/using-scrapy-from-a-script/

我不知道应该把蜘蛛代码放在哪里以及如何从主函数中调用它。请帮忙。这是示例代码：

# This snippet can be used to run scrapy spiders independent of scrapyd or the scrapy command line tool and use it from a script. 
# 
# The multiprocessing library is used in order to work around a bug in Twisted, in which you cannot restart an already running reactor or in this case a scrapy instance.
# 
# [Here](http://groups.google.com/group/scrapy-users/browse_thread/thread/f332fc5b749d401a) is the mailing-list discussion for this snippet. 

#!/usr/bin/python
import os
os.environ.setdefault('SCRAPY_SETTINGS_MODULE', 'project.settings') #Must be at the top before other imports

from scrapy import log, signals, project
from scrapy.xlib.pydispatch import dispatcher
from scrapy.conf import settings
from scrapy.crawler import CrawlerProcess
from multiprocessing import Process, Queue

class CrawlerScript():

    def __init__(self):
        self.crawler = CrawlerProcess(settings)
        if not hasattr(project, 'crawler'):
            self.crawler.install()
        self.crawler.configure()
        self.items = []
        dispatcher.connect(self._item_passed, signals.item_passed)

    def _item_passed(self, item):
        self.items.append(item)

    def _crawl(self, queue, spider_name):
        spider = self.crawler.spiders.create(spider_name)
        if spider:
            self.crawler.queue.append_spider(spider)
        self.crawler.start()
        self.crawler.stop()
        queue.put(self.items)

    def crawl(self, spider):
        queue = Queue()
        p = Process(target=self._crawl, args=(queue, spider,))
        p.start()
        p.join()
        return queue.get(True)

# Usage
if __name__ == "__main__":
    log.start()

    """
    This example runs spider1 and then spider2 three times. 
    """
    items = list()
    crawler = CrawlerScript()
    items.append(crawler.crawl('spider1'))
    for i in range(3):
        items.append(crawler.crawl('spider2'))
    print items

# Snippet imported from snippets.scrapy.org (which no longer works)
# author: joehillen
# date  : Oct 24, 2010

谢谢。

Answer 1

所有其他答案均参考 Scrapy v0.x。根据the updated docs，Scrapy 1.0 要求：

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

Answer 2

我们可以使用

from scrapy.crawler import CrawlerProcess
from project.spiders.test_spider import SpiderName

process = CrawlerProcess()
process.crawl(SpiderName, arg1=val1,arg2=val2)
process.start()

在具有全局范围的蜘蛛 __init__ 函数中使用这些参数。

Answer 3

虽然我没有尝试过，但我认为答案可以在scrapy documentation 中找到。直接引用它：

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy.settings import Settings
from scrapy import log
from testspiders.spiders.followall import FollowAllSpider

spider = FollowAllSpider(domain='scrapinghub.com')
crawler = Crawler(Settings())
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run() # the script will block here

据我所知，这是图书馆的一项新发展，它使一些早期的在线方法（例如问题中的方法）过时了。

Answer 4

在scrapy 0.19.x 中你应该这样做：

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from testspiders.spiders.followall import FollowAllSpider
from scrapy.utils.project import get_project_settings

spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run() # the script will block here until the spider_closed signal was sent

注意这几行

settings = get_project_settings()
crawler = Crawler(settings)

没有它，您的蜘蛛将不会使用您的设置，也不会保存项目。 我花了一段时间才弄清楚为什么文档中的示例没有保存我的项目。我发送了一个拉取请求来修复文档示例。

另外一个方法就是直接从你的脚本中调用命令

from scrapy import cmdline
cmdline.execute("scrapy crawl followall".split())  #followall is the spider's name

从我在这里的第一个答案中复制了这个答案： https://stackoverflow.com/a/19060485/1402286

Answer 5

当需要在一个python脚本中运行多个爬虫时，需要谨慎处理reactor停止，因为reactor只能停止一次，不能重新启动。

但是，我在做我的项目时发现使用

os.system("scrapy crawl yourspider")

是最简单的。这将使我免于处理各种信号，尤其是当我有多个蜘蛛时。

如果性能是一个问题，您可以使用多处理来并行运行您的蜘蛛，例如：

def _crawl(spider_name=None):
    if spider_name:
        os.system('scrapy crawl %s' % spider_name)
    return None

def run_crawler():

    spider_names = ['spider1', 'spider2', 'spider2']

    pool = Pool(processes=len(spider_names))
    pool.map(_crawl, spider_names)

Answer 6

这是一个改进 Scrapy throws an error when run using crawlerprocess

和https://github.com/scrapy/scrapy/issues/1904#issuecomment-205331087

首先创建您常用的爬虫以成功运行命令行。它应该运行并导出数据或图像或文件非常重要

一旦结束，就像在我的程序中粘贴蜘蛛类定义上方和 __name __ 下方一样调用设置。

它将获得许多人推荐的“from scrapy.utils.project import get_project_settings”未能执行的必要设置

上面和下面的部分应该在一起。只有一个不跑。 Spider 将在 scrapy.cfg 文件夹中运行，而不是在任何其他文件夹中运行

版主可以展示树状图供参考

#Tree
[enter image description here][1]

#spider.py
import sys
sys.path.append(r'D:\ivana\flow') #folder where scrapy.cfg is located

from scrapy.crawler import CrawlerProcess
from scrapy.settings import Settings
from flow import settings as my_settings

#----------------Typical Spider Program starts here-----------------------------

          spider class definition here

#----------------Typical Spider Program ends here-------------------------------

if __name__ == "__main__":

    crawler_settings = Settings()
    crawler_settings.setmodule(my_settings)

    process = CrawlerProcess(settings=crawler_settings)
    process.crawl(FlowSpider) # it is for class FlowSpider(scrapy.Spider):
    process.start(stop_after_crawl=True)

Answer 7

# -*- coding: utf-8 -*-
import sys
from scrapy.cmdline import execute


def gen_argv(s):
    sys.argv = s.split()


if __name__ == '__main__':
    gen_argv('scrapy crawl abc_spider')
    execute()

将此代码放在您可以从命令行运行scrapy crawl abc_spider 的路径中。 （用 Scrapy 测试==0.24.6）

Answer 8

如果你想进行简单的爬取，只需运行命令即可：

爬虫。 还有另一种选择可以将结果导出为以某些格式存储，例如： json、xml、csv。

scrapy crawl -o result.csv 或 result.json 或 result.xml。

你可以试试