Python - 运行几个小时后的 Selenium webdriver 错误

Question

我在云机器（AWS 1GB 实例）中运行 BOT，我正在使用 selenium-webdriver + pyvirtualdisplay。

我在一个循环中做一些重复的事情，这将迭代 50000 次。运行我的脚本后，它在第 1016 个停止工作（共 50000 个）。错误是

Traceback (most recent call last):
  File "automation.py", line 108, in <module>
    main()
  File "automation.py", line 10, in main
    driver = webdriver.Firefox()
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.binary, timeout),
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 61, in launch_browser
    self._wait_until_connectable()
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 105, in _wait_until_connectable
    self.profile.path, self._get_firefox_output()))
selenium.common.exceptions.WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmp7CXXn1 Firefox output: \n(process:22693): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1213".\n1406467599310\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1406467599312\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1406467599314\taddons.xpi\tDEBUG\tstartup\n1406467599339\taddons.xpi\tDEBUG\tcheckForChanges\n1406467599351\taddons.xpi\tDEBUG\tNo changes found\nSystem JS : ERROR chrome://browser/content/browser.js:12132 - SyntaxError: function statement requires a name\n 
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n' 

From error 好像是内存有问题？是吗？ 不知道怎么解决？

事实上，如果我重新运行这个脚本它没有，它会显示这个错误。

selenium.common.exceptions.WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: \n(process:22882): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1295".\n1406470964822\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1406470964823\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1406470964826\taddons.xpi\tDEBUG\tstartup\n1406470964847\taddons.xpi\tDEBUG\tcheckForChanges\n1406470964855\taddons.xpi\tDEBUG\tNo changes found\n************************************************************\n* Call to xpconnect wrapped JSObject produced this error:  *\n[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///tmp/tmpdCliUY/extensions/fxdriver@googlecode.com/components/bad_cert_listener.js :: WdCertOverrideService :: line 3855"  data: no]\n************************************************************\n************************************************************\n* Call to xpconnect wrapped JSObject produced this error:  *\n[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///tmp/tmpdCliUY/extensions/fxdriver@googlecode.com/components/bad_cert_listener.js :: WdCertOverrideService :: line 3855"  data: no]\n************************************************************\nJavaScript error: chrome://fxdriver/content/server.js, line 38: NS_ERROR_XPC_CI_RETURNED_FAILURE: Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE) [nsIJSCID.createInstance]\n\n(firefox:22882): LIBDBUSMENU-GLIB-WARNING **: Unable to get session bus: Failed to fork (Cannot allocate memory)\nout of memory: 0x0000000000040028 bytes requested\n' 

我的代码如下所示，供您参考。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from pyvirtualdisplay import Display

display = Display(visible=0, size=(1024, 768))
display.start()
driver = webdriver.Firefox()
driver.maximize_window()

url = 'some url'

for i in range(1,50000):
    driver.get(url)
    #somestuff
    i+=1

Answer 1

从错误日志看来您使用的是 unix 风格

我过去也犯过同样的错误。 很明显我们可以看到系统无法提供足够的内存。

运行top 命令并查看可用内存是否足够。如果没有，则杀死一些不必要的进程。

这应该可行。