无法安装 MySQLdb-python==1.2.5 Scrapinghub (Scrapy) Python 2.7

Question

我阅读了一些关于连接 Mysql 和 scrapinghub 部署脚本的线程。 他们建议更改 *.yml 文件并添加需求 txt。 该解决方案几天前有效。现在不行了。

这是来自 Shub Deploy 的错误。

 Collecting MySQLdb-python==1.2.5 (from -r /app/requirements.txt (line 1))
←[91m  Could not find a version that satisfies the requirement MySQLdb-python==1.2.5 (from -r /app/
requirements.txt (line 1)) (from versions: )
←[0m
←[91mNo matching distribution found for MySQLdb-python==1.2.5 (from -r /app/requirements.txt (line
1))
←[0m
{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip insta
ll --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"messa
ge": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --us
er --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "req
uirements_error"}
{"status": "error", "message": "Requirements error"}

是 SH 改变了 smth 还是我哪里错了？

Answer 1

1) 你应该在你的requirements.txt 文件中写MySQL-python==1.2.5，这是我所拥有的，我的项目运行良好。

然后我像这样使用MYSQL

import MySQLdb

conn = MySQLdb.connect(user=DB_CREDS['user'], passwd=DB_CREDS['pass'], db=DB_CREDS['db'], host=DB_CREDS['host'], charset="utf8", use_unicode=True)

cursor = MySQLdb.cursors.DictCursor(conn) 

conn.autocommit(True)

cursor.execute("SELECT myColumn FROM table_name")

for row in self.cursor.fetchall():
    print row['myColumn']

2) 在再次部署之前，还要删除 build/ 和 project.egg-info/ 文件夹。

3) 确保您的 scrapinghub.yml 文件看起来像这样

projects:
  default: 123213213
requirements_file: requirements.txt

Answer 2

遇到同样的问题。

蟒蛇：3.9.6 点数：21.1.3 shub: 2.13.0

// scrapyinghub.yml
projects:
  default: 70449

requirements_file: requirements.txt

// requirements.txt
pip==21.1.3

$ python -m shub deploy
Deploying to Scrapy Cloud project "70449"
Deploy log last 30 lines:
 ---> Using cache
 ---> d5f56e9b2c2f
Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/
 ---> Using cache
 ---> e4326e357a5c
Step 4/12 : ADD run-pipcheck /usr/local/bin/
 ---> Using cache
 ---> 72567751d2ac
Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck
 ---> Using cache
 ---> 365b8f799cc4
Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck
 ---> Using cache
 ---> 3b80a05baecd
Step 7/12 : ADD requirements.txt /app/requirements.txt
 ---> Using cache
 ---> 81a2d0e0fc10
Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python
 ---> Using cache
 ---> 12d562276a09
Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE -E PIP_NO_CACHE_DIR=0     pip install --user --no-cache-dir -r /app/requirements.txt
 ---> Running in 0c8c5e436aea
Collecting pip==21.1.3 (from -r /app/requirements.txt (line 1))
  Could not find a version that satisfies the requirement pip==21.1.3 (from -r /app/requirements.txt (line 1)) (from versions: )

No matching distribution found for pip==21.1.3 (from -r /app/requirements.txt (line 1))

{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE -E PIP_NO_CACHE_DIR=0     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE -E PIP_NO_CACHE_DIR=0     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /tmp/shub_deploy_se518dkv.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'

更新

指定scrapy-stack然后部署成功。

projects:
  default: 70449

stack: scrapy:1.3-py3

requirements_file: requirements.txt

$ python -m shub deploy

Packing version v1.0.1-7-g00c7efa-tmp
Deploying to Scrapy Cloud project "70449"
{"status": "ok", "project": 70449, "version": "v1.0.1-7-g00c7efa-tmp", "spiders": 3}
Run your spiders at: https://app.scrapinghub.com/p/70449/