Apscheduler+scrapy 信号只在主线程有效

【问题标题】：Apscheduler+scrapy signal only works in main threadApscheduler+scrapy 信号只在主线程有效
【发布时间】：2018-12-04 02:53:29
【问题描述】：

我想将 apscheduler 与 scrapy 结合起来。但我的代码是错误的。应该怎么修改？

settings = get_project_settings()
configure_logging(settings)
runner = CrawlerRunner(settings)

@defer.inlineCallbacks
def crawl():
    reactor.run()
    yield runner.crawl(Jobaispider)#this is my spider
    yield runner.crawl(Jobpythonspider)#this is my spider
    reactor.stop()

sched = BlockingScheduler()
sched.add_job(crawl, 'date', run_date=datetime(2018, 12, 4, 10, 45, 10))
sched.start()

Error:builtins.ValueError: 信号只在主线程中起作用

【问题讨论】：

标签： scrapy apscheduler

【解决方案1】：

这个问题已经在这里得到了很好的回答：How to integrate Flask & Scrapy?，它涵盖了各种用例和想法。我还发现该线程中的一个链接非常有用：https://github.com/notoriousno/scrapy-flask

要更直接地回答您的问题，请尝试一下。它使用了上面两个链接的解决方案，特别是它使用了钩针库。

import crochet
crochet.setup()

settings = get_project_settings()
configure_logging(settings)
runner = CrawlerRunner(settings)

# Note: Removing defer here for the example
#@defer.inlineCallbacks

@crochet.run_in_reactor
def crawl():
    runner.crawl(Jobaispider)#this is my spider
    runner.crawl(Jobpythonspider)#this is my spider

sched = BlockingScheduler()
sched.add_job(crawl, 'date', run_date=datetime(2018, 12, 4, 10, 45, 10))
sched.start()

【讨论】：