为什么scrapy shell不适用于此网址？答案

【问题标题】：Why is scrapy shell not working for this url?为什么scrapy shell不适用于此网址？
【发布时间】：2020-12-02 03:09:23
【问题描述】：

我正在使用 scrapy，python 的网络抓取库来抓取一些数据。我在其他页面上使用了“scrapy shell url”，但由于某种原因它不适用于下面的 url

https://feedback.aliexpress.com/display/productEvaluation.htm?&productId=4000033626141&ownerMemberId=22

scrapy shell 应该会产生一个“>>>”，然后你就可以使用响应对象了。当使用完全相同的蜘蛛进行刮壳时，但使用上面的链接，它不会产生任何结果。下面是输出

scrapy shell https://feedback.aliexpress.com/display/productEvaluation.htm?&productId=4000033626141&ownerMemberId=22
[12] 40068
[13] 40069
[10]   Exit 1                  scrapy shell https://feedback.aliexpress.com/display/productEvaluation.htm?
[11]   Done                    productId=4000033626141
(aliexpress_scraping) (base) othos-Air:aliexpress_scraping otho$ 2020-12-01 22:08:21 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: googleweblight)
2020-12-01 22:08:21 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.6.0 (default, Nov  1 2020, 22:24:33) - [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)], pyOpenSSL 20.0.0 (OpenSSL 1.1.1h  22 Sep 2020), cryptography 3.2.1, Platform Darwin-18.7.0-x86_64-i386-64bit
2020-12-01 22:08:21 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-12-01 22:08:21 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'googleweblight',
 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter',
 'LOGSTATS_INTERVAL': 0,
 'NEWSPIDER_MODULE': 'aliexpress_scraper.spiders',
 'SPIDER_MODULES': ['aliexpress_scraper.spiders']}
2020-12-01 22:08:21 [scrapy.extensions.telnet] INFO: Telnet Password: 642763dfc6c8dd26
2020-12-01 22:08:21 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage']
2020-12-01 22:08:21 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-12-01 22:08:21 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-12-01 22:08:21 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-12-01 22:08:22 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6030
2020-12-01 22:08:22 [scrapy.core.engine] INFO: Spider opened
2020-12-01 22:08:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://feedback.aliexpress.com/display/productEvaluation.htm> (referer: None)

【问题讨论】：

标签： python python-3.x web-scraping scrapy

【解决方案1】：

该 URL 包含在某些系统 shell（例如 Bash）中属于特殊字符的字符（例如 &）。

如果你使用的是 Linux shell，如果你把 URL 放在单引号之间，事情可能会奏效：

$ scrapy shell 'https://feedback.aliexpress.com/display/productEvaluation.htm?&productId=4000033626141&ownerMemberId=22'

或者，一旦进入 Scrapy shell，您可以使用 fetch(<URL as a Python string>) 加载不同的 URL。

【讨论】：