当源 URL 不是状态 200 时脚本停止答案

【问题标题】：Script STOPS when Source URL is not Status 200当源 URL 不是状态 200 时脚本停止
【发布时间】：2013-06-02 20:40:00
【问题描述】：

我有一个脚本（见下文）可以检查（或多或少）网站上的链接位置，它工作正常，但只要链接所在的源 URL 不是 200 响应，它就会退出，我只想它可以跳到下一个或返回一些消息“错误”，甚至更好地返回 http 状态代码。我需要一个快速的解决方案，如果有人可以帮助我，那就太棒了:)

URLs.csv = 包含指向某个页面的链接的网站列表
domain.com = 用于检查链接是否存在的域，如果存在，则它大致位于何处。

import csv
from lxml import html

with open('URLs.csv', 'r') as csvfile:
    urls = [row[0] for row in csv.reader(csvfile)]

for url in urls:
    print url

    doc = html.parse(url)
    if doc.xpath('//a[contains(@href,"domain.com")]'):
        for anchor_node in doc.xpath('//a[contains(@href,"finanzen.de")]'):
            if anchor_node.xpath('./ancestor::div[contains(@class, "sidebar")]'):
                print 'Sidebar'
            elif anchor_node.xpath('./parent::div[contains(@class, "widget")]'):
                print 'Sidebar'
            elif anchor_node.xpath('./ancestor::div[contains(@id, "sidebar")]'):
                print 'Sidebar'            
            elif anchor_node.xpath('./ancestor::div[contains(@class, "comment")]'):
                print 'Kommentar'
            elif anchor_node.xpath('./ancestor::div[contains(@id, "comment")]'):
                print 'Kommentar'
            elif anchor_node.xpath('./ancestor::div[contains(@class, "foot")]'):
                print "Footer"
            elif anchor_node.xpath('./ancestor::div[contains(@id, "foot")]'):
                print "Footer"
            elif anchor_node.xpath('./ancestor::div[contains(@class, "post")]'):
                print "Contextual"
            else:
                print 'Unidentified Link'
        else:
            print 'Link is Dead'

Python-Shell

            Python 2.7.4 (default, Apr  6 2013, 19:55:15) [MSC v.1500 64 bit (AMD64)]
            Type "help", "copyright", "credits" or "license" for more information.
            [evaluate Linkidentifizierung.py]
            http://urlnotworking.com/broken.html
            Rückverfolgung (innerste zuletzt):
            File "C:\Program Files (x86)\Wing IDE 101 4.1\src\debug\tserver\_sandbox.py",     line 11, in <module>
            File "C:\Python27\Lib\site-packages\lxml\html\__init__.py", line 735, in parse
        return etree.parse(filename_or_url, parser, base_url=base_url, **kw)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 3197, in  lxml.etree.parse (src\lxml\lxml.etree.c:64726)
       H‹GH‹ÏÿP0H…ÛtHƒÿu
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1571, in lxml.etree._parseDocument (src\lxml\lxml.etree.c:92363)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1600, in lxml.etree._parseDocumentFromURL (src\lxml\lxml.etree.c:92647)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1500, in lxml.etree._parseDocFromFile (src\lxml\lxml.etree.c:91710)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 1047, in lxml.etree._BaseParser._parseDocFromFile (src\lxml\lxml.etree.c:88610)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:84019)
            File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 676, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:85122)
           File "C:\Python27\Lib\site-packages\lxml\etree.pyd", line 614, in lxml.etree._raiseParseError (src\lxml\lxml.etree.c:84417)
    IOError: Error reading file 'http://urlnotworking.com/broken.html':  failed to load HTTP resource

【问题讨论】：

我必须修正你的缩进；您现在可以验证一切是否正确吗？
谢谢，我在这里仍然很新 :) 但我的问题仍然存在，只要每个 URL 似乎都有 200 响应，它就可以工作，如果没有，它会生成 python-shell，如上所示。

标签： python xml parsing xpath web-scraping

【解决方案1】：

print url

try:
  html.parse(url)
except Exception, e:
  print "something went wrong: %s" % e
  continue

if doc.xpath....

我对 lxml.html 库不够熟悉，因此无法了解有关 URL 加载失败原因的更详细信息。（提示——我会使用“请求”库来加载我的 url，然后将结果传递给 lxml 解析器。）

【讨论】：