Scrapy：错误：蜘蛛错误处理

Question

我是 python 和 scrapy 的新手。 我尝试运行现有代码，但每个地址都出现此错误：

>     2015-07-02 01:52:19 [scrapy] DEBUG: Crawled (200) <GET http://www.tripadvisor.com/ShowUserReviews-g187147-d197524-r281927613-Hotel_Mirific_Opera-Paris_Ile_de_France.html>
> (referer:
> http://www.tripadvisor.com/Hotel_Review-g187147-d197524-Reviews-Hotel_Mirific_Opera-Paris_Ile_de_France.html)2015-07-02
> 01:52:19 
>     [scrapy] ERROR: Spider error processing <GET http://www.tripadvisor.com/ShowUserReviews-g187147-d197524-r281927613-Hotel_Mirific_Opera-Paris_Ile_de_France.html>
> (referer:
> http://www.tripadvisor.com/Hotel_Review-g187147-d197524-Reviews-Hotel_Mirific_Opera-Paris_Ile_de_France.html)
> 
    > Traceback (most recent call last):   File
    > "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line
    > 102, in iter_errback
    >     yield next(it)   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py",
    > line 28, in process_spider_output
    >     for x in result:   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py",
    > line 22, in <genexpr>
    >     return (_set_referer(r) for r in result or ())   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py",
    > line 37, in <genexpr>
    >     return (r for r in result or () if _filter(r))   File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py",
    > line 54, in <genexpr>
    >     return (r for r in result or () if _filter(r))   File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/crawl.py", line
    > 67, in _parse_response
    >     cb_res = callback(response, **cb_kwargs) or ()   File "/home/talmosko/Documents/scrapy/tripAdvisor/spiders/tripAdvisor.py",
    > line 30, in parse_item
    >      item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()[0].encode('ascii',
    > errors='ignore')
    > 
    > IndexError: list index out of range

这是我的代码： http://pastebin.com/XzM5DrDD

有什么问题？蜘蛛似乎没有得到答案..

谢谢！

Answer 1

你正在尝试访问一个不存在的元素，错误就在这一行

item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()[0].encode('ascii', errors='ignore')

可能

item['state'] =  hxs.xpath('//*[@id="PAGE"]/div[2]/div[1]/ul/li[2]/a/span/text()').extract()

为空，您正在尝试访问第一个元素。你有两个选择：

• 修改选择器使其始终返回一些数据，使用Scrapy shell 测试它可能是个好主意
• Try and catch the IndexError