从脚本运行scrapy时能够更改设置答案

【问题标题】：Being able to change the settings while running scrapy from a script从脚本运行scrapy时能够更改设置
【发布时间】：2015-10-13 04:45:37
【问题描述】：

我想run scrapy from a single script 并且我想从 settings.py 中获取所有设置，但我希望能够更改其中的一些设置：

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

*### so what im missing here is being able to set or override one or two of the settings###*


# 'followall' is the name of one of the spiders of the project.
process.crawl('testspider', domain='scrapinghub.com')
process.start() # the script will block here until the crawling is finished

我无法使用this。我尝试了以下方法：

settings=scrapy.settings.Settings()
settings.set('RETRY_TIMES',10)

但是没有用。

注意：我使用的是最新版本的scrapy。

【问题讨论】：

标签： python scrapy

【解决方案1】：

因此，为了覆盖某些设置，一种方法是在我们的脚本中覆盖/设置 custom_settings，即蜘蛛的静态变量。

所以我导入了蜘蛛的类，然后覆盖了 custom_setting：

from testspiders.spiders.followall import FollowAllSpider 

FollowAllSpider.custom_settings={'RETRY_TIMES':10}

这就是整个脚本：

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from testspiders.spiders.followall import FollowAllSpider 

FollowAllSpider.custom_settings={'RETRY_TIMES':10}
process = CrawlerProcess(get_project_settings())


# 'followall' is the name of one of the spiders of the project.
process.crawl('testspider', domain='scrapinghub.com')
process.start() # the script will block here until the crawling is finished

【讨论】：

【解决方案2】：

由于某种原因，上面的脚本对我不起作用。相反，我写了以下内容并且它有效。发帖以防其他人遇到同样的问题。

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.settings.set(
            'RETRY_TIMES', 10, priority='cmdline')

process.crawl('testspider', domain='scrapinghub.com')
process.start()

【讨论】：

【解决方案3】：

我自己遇到了这个问题，并有一个稍微不同的解决方案，它使用现代 Python (>=3.5) 方法

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

settings = {
    **get_project_settings(),
    'RETRY_TIMES': 2
}


process = CrawlerProcess(settings)
process.crawl('testspider', domain='scrapinghub.com')
process.start()

【讨论】：