【发布时间】:2017-02-16 17:10:57
【问题描述】:
我尝试在 Ubuntu 上使用 crontab 运行 python scrapy 爬虫,但收到以下错误消息:
Traceback (most recent call last):
File "/usr/bin/scrapy", line 9, in <module>
load_entry_point('Scrapy==1.0.3', 'console_scripts', 'scrapy')()
File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 142, in execu$
cmd.crawler_process = CrawlerProcess(settings)
File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 209, in __ini$
super(CrawlerProcess, self).__init__(settings)
File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 115, in __ini$
self.spider_loader = _get_spider_loader(settings)
File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 296, in _get_$
return loader_cls.from_settings(settings.frozencopy())
File "/usr/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 30, in f$
return cls(settings)
File "/usr/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 21, in _$
for module in walk_modules(name):
File "/usr/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 71, in wal$
submod = import_module(fullpath)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/kebodev/scrapy/qgtest2/qgtest2/spiders/jsonspider.py", line 5, in <module>
import cx_Oracle
ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory
我用 root 用户编辑我的 ~/.bashrc 并添加了以下行:
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export ORACLE_SID=XE
export NLS_LANG=`$ORACLE_HOME/bin/nls_lang.sh`
export ORACLE_BASE=/u01/app/oracle
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH
我的 libclntsh.so.11.1 位于此处:/u01/app/oracle/product/11.2.0/xe/lib
如果我尝试从终端运行我的 python scrapy 爬虫,它正在运行,如果我尝试在 python shell 中导入 cx_Oracle,它也可以工作,但使用 crontab 就不行了..
这就是我的 cron 工作线的样子:
* * * * * root /etc/listarunner.sh >> /home/kebodev/scrapy/qgtest2/etcronlog1.log 2>&1
这是我的listarunner.sh 文件:
#!/bin/bash
cd /home/kebodev/scrapy/qgtest2
PATH=$PATH:/usr/local/bin
export PATH
scrapy crawl jsontst
谁能帮帮我?
谢谢!
【问题讨论】:
标签: python ubuntu scrapy crontab cx-oracle