尝试检索图像链接时出现 Python 错误

Question

每次运行以下 Python 脚本时，我都会收到错误 AttributeError: 'NoneType' object has no attribute 'findAll'。我做了一些研究，发现一些帖子表明我在尝试查找图像时可能会传递“无”，这就是它出错的原因。我仍然没有解决方案。任何信息都有帮助。

这是完整的错误：

Traceback (most recent call last):
  File "D:\Program Files\Parser Python\Test.py", line 33, in <module>
    for img in divImage.findAll('img'):
AttributeError: 'NoneType' object has no attribute 'findAll'


    from bs4 import BeautifulSoup

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.common.exceptions import TimeoutException
import os

firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
firefox_capabilities['binary'] = 'C:\Program Files (x86)\Mozilla Firefox\firefox.exe'


os.environ["PATH"] += "C:\Python27\Lib\site-packages\selenium-2.53.6-py2.7.egg\selenium"
#binary = FirefoxBinary('C:\Program Files (x86)\Mozilla Firefox\firefox.exe')
driver = webdriver.Firefox(capabilities=firefox_capabilities)
# it takes forever to load the page, therefore we are setting a threshold
driver.set_page_load_timeout(5)

try:
    driver.get("http://readcomiconline.to/Comic/Flashpoint/Issue-1?id=19295&readType=1")
except TimeoutException:
    # never ignore exceptions silently in real world code
    pass

soup2 = BeautifulSoup(driver.page_source, 'html.parser')
divImage = soup2.find('div', {"id": "divImage"})
#divImage = soup2.find('div', {"id": "containerRoot"})

# close the browser 
driver.close()

for img in divImage.findAll('img'):
    print img.get('src')

Answer 1

该错误表示divImage是None，这意味着在解析的HTML中没有找到带有id="divImage"的div元素。

您应该首先等待页面上出现所需的元素，然后才能获取页面源并对其进行解析。这可以通过WebDriverWait 来完成：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ...

driver.get("http://readcomiconline.to/Comic/Flashpoint/Issue-1?id=19295&readType=1")

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.ID, "divImage")))

soup2 = BeautifulSoup(driver.page_source, 'html.parser')

注意，要等待所有图片加载完毕，需要不断滚动页面到页脚，直到所有图片都加载完毕，实现：

driver.get("http://readcomiconline.to/Comic/Flashpoint/Issue-1?id=19295&readType=1")
wait.until(EC.presence_of_element_located((By.ID, "divImage")))

footer = driver.find_element_by_id("footer")

while True:
    # scroll to the footer
    driver.execute_script("arguments[0].scrollIntoView();", footer)
    time.sleep(0.5)

    # check if all images are loaded
    if all(img.get_attribute("src") for img in driver.find_elements_by_css_selector("#divImage p img")):
        break

别忘了import time。