Scrapinghub scrapy: ModuleNotFoundError: No module named 'pandas'

Question

我曾尝试通过命令行和 GitHub 部署到 Zyte，但遇到上述错误。

我尝试了不同版本的 Scrapy 1.5 到 2.5，但错误仍然存​​在。

我也尝试将我的 Scrapinghub.yml 设置为以下脚本，但似乎无法解决错误

project: 557222

stacks:
    default: scrapy:1.5-py3






 Deploy log location: C:\Users\Public\Documents\Wondershare\CreatorTemp\shub_deploy_xzbjx9qc.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'
  File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy
    execute(settings=settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 249, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 336, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/usr/local/lib/python3.6/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/app/__main__.egg/greyhound_recorder_website/spiders/greyhound.py", line 7, in <module>
ModuleNotFoundError: No module named 'pandas'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}
{"status": "error", "message": "Internal error"}

Answer 1

您应该指定 dependencies you need to be installed in Scrapy Cloud 。例如：

scrapinghub.yml：

project: 557222

stacks:
    default: scrapy:1.5-py3

requirements:
    file: requirements.txt

requirements.txt：

pandas

Answer 2

Pandas 是一个类似于 Scrapy 的 python 包。你需要安装它。 pip install pandas 或访问https://pandas.pydata.org/docs/getting_started/install.html