httplib.BadStatusLine: '' 在 Linux 但不是 Mac

【问题标题】：httplib.BadStatusLine: '' on Linux but not Machttplib.BadStatusLine: '' 在 Linux 但不是 Mac
【发布时间】：2017-04-09 13:01:33
【问题描述】：

这个错误已经困扰我几个小时了。我决定编写一个单独的项目，看看我是否可以复制它，我可以，但只能在我的服务器上。这适用于我的 Mac。

  • Mac：OSX El Capitan 10.11.6

  • 服务器：CentOS 7.2.1511

  • 两者都有 PhantomJS 版本：2.1.1

  • Python Mac：Python 2.7.11

  • Python 服务器：2.7.5

  • 两者都有 Selenium 版本：2.53.0

两者运行相同的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.common.exceptions import NoSuchElementException
import time

dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36"
dcap["phantomjs.page.customHeaders.accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
dcap["phantomjs.page.customHeaders.Accept-Language"] = "en-US,en;q=0.8"
dcap["phantomjs.page.customHeaders.connection"] = "keep-alive"

driver = webdriver.PhantomJS(desired_capabilities=dcap)
driver.set_window_size(1120, 700)
driver.get("https://www.instagram.com/espn/")

while True:
    print len(driver.find_elements_by_css_selector("a[href*='/p/']"))
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        loadMore = driver.find_element_by_link_text("Load more")
        loadMore.click()
    except NoSuchElementException:
        print "No such"
        driver.save_screenshot('none.png')

Mac 输出：

12
24
No such
24
No such
36
No such
48
No such
48
No such
60
No such
72
No such
84
# This goes until I end it

服务器输出：

12
24
No such
Traceback (most recent call last):
  File "junk.py", line 27, in <module>
    driver.save_screenshot('none.png')
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 790, in get_screenshot_as_file
    png = self.get_screenshot_as_png()
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 809, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 819, in get_screenshot_as_base64
    return self.execute(Command.SCREENSHOT)['value']
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

删除截图行后的服务器输出：

12
24
No such
24
Traceback (most recent call last):
  File "junk.py", line 23, in <module>
    loadMore = driver.find_element_by_link_text("Load more")
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in find_element_by_link_text
    return self.find_element(by=By.LINK_TEXT, value=link_text)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 744, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1244, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1217, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 444, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 408, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

我在这里找到了一个相关的答案：Can't run PhantomJS in python via Selenium

所以我安装了 Selenium 2.37，它给出了同样的错误。

我阅读了this answer 关于可能与更改标头有关的问题，因此我通过将驱动程序更改为driver = webdriver.PhantomJS() 来删除标头，但仍然遇到相同的错误。

我也在服务器上安装了2.7.12，看看有没有区别。输出是：

# python2.7 junk.py
12
24
No such
24
Traceback (most recent call last):
  File "junk.py", line 29, in <module>
    loadMore = driver.find_element_by_link_text("Load more")
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 314, in find_element_by_link_text
    return self.find_element(by=By.LINK_TEXT, value=link_text)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 744, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
    response = self.command_executor.execute(driver_command, params)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
    return self._request(command_info[0], url, body=data)
  File "/usr/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 463, in _request
    resp = opener.open(request, timeout=self._timeout)
  File "/usr/local/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/local/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/local/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/local/lib/python2.7/httplib.py", line 1136, in getresponse
    response.begin()
  File "/usr/local/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python2.7/httplib.py", line 417, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

检查系统空间。这是一个全新的 VPS，但仍然需要确认：

【问题讨论】：

  • 这里有一些有用的信息stackoverflow.com/a/27620850/2575259
  • 您可以看到异常跟踪，WebDriver 方法（find_element_by_link_text）如何被视为命令并构建 HTTP 请求并发送到特定的 WebDriver 实现（此处为 PhantomJs）并通过获取完成工作HTTP 响应返回。在您的情况下，该 http 请求返回的 HTTP 状态代码是 WebDriver 的客户端绑定无法理解的
  • 根据长期经验，我发现如果我在滚动到后立即睡 500 毫秒，效果最好。假设是当您尝试保存屏幕时 scrollTo 仍在执行。所以试着把延迟放进去，看看它是否会改变行为。如果不是这样，我们就会开始产生更多假设。
  • 假设3：与保存文件的默认位置有关。请将您的 driver.save_screenshot('none.png') 替换为 driver.get_screenshot_as_file('/tmp/none.png')。
  • 您能否总结此评论线程中的相关要点以及先前接受的答案并更新您的问题。目前，由于 cmets 线程很长，很难真正理解已尝试过和未尝试过的内容等

标签： python selenium selenium-webdriver centos phantomjs


【解决方案1】：

编辑 3

添加以下内容：

except httplib.BadStatusLine:
    pass

编辑 2

Python WebDriver 和 phantomJs 的 keep_alive 有问题。这可能是你的问题。所以添加keep_alive=False如下：

driver = webdriver.PhantomJS(desired_capabilities=dcap,keep_alive=False)

结束编辑

添加以下内容

import httplib
import socket

from selenium.webdriver.remote.command import Command

def get_status(driver):
    try:
        driver.execute(Command.STATUS)
        return "Alive"
    except (socket.error, httplib.CannotSendRequest):
        return "Dead"

在 save_screenshot 语句之前调用 get_status(driver) 并打印结果。这将告诉我们驱动程序是否过早关闭。

编辑

在 driver = webdriver.PhantomJS(desired_capabilities=dcap) 之后添加以下内容

driver.implicitly_wait(10) #wait 10 seconds when doing a find_element before carrying on

【讨论】：

  • 有趣。所以它在print get_status(driver)上给出httplib.BadStatusLine: ''
  • 驱动程序死了，它不能告诉我们它已经死了？如果您在 driver.get("instagram.com/espn/") 之后立即调用 get_status 会怎样
  • 第一行是Alive，然后是常规的。
  • @User 我编辑了答案以提出建议。如果失败了，那么如果与编辑的更改一起在 loadMore = driver.find_element_by_link_text("Load more") 之前插入对 get_status(driver) 的调用，会发生什么？
  • 在 print len(find elements) 之后和滚动之前添加它。输出是：Alive 12 24 然后它在driver.find_element_by_link_text("Load more") 上抛出了同样的异常
猜你喜欢
  • 2015-02-21
  • 2019-10-15
  • 2013-06-23
  • 2015-12-19
  • 1970-01-01
  • 2016-01-06
  • 1970-01-01
  • 2020-10-22
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode