学习使用 Selenium 和 Python 进行抓取

Question

我正在学习使用 selenium 进行刮擦，但无法连接到此站点“http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720”

它不会加载网站的内容

我想学习如何连接到此站点以请求图像和数据

我的代码很简单，因为我正在学习，我寻找建立连接的方法但没有成功

from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

ff_profile = FirefoxProfile()
ff_profile.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36")
driver = webdriver.Firefox(firefox_profile = ff_profile)
driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
time.sleep(5)
campo_busca = driver.find_elements_by_id('of132')
print(campo_busca)

Answer 1

由于所需元素位于 <iframe> 内，因此要调用提取所需元素的 src 属性，您必须：

• 诱导 WebDriverWait 使所需的框架可用并切换到它。
• 为所需的visibility_of_element_located() 诱导 WebDriverWait。

• 您可以使用以下Locator Strategies：

  driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
  driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
  WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[@id='CamosIFId' and @name='CamosIF']")))
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//img[@id='of132']"))).get_attribute("src"))
  

• 然而，正如 @google 提到的 cmets 之一，使用 ChromeDriver / Chrome 的浏览体验似乎更好，您可以使用以下解决方案：

  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('http://www.festo.com/cat/it_it/products_VUVG_S?CurrentPartNo=8043720')
  WWebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#CamosIFId[name='CamosIF']")))
  print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "img#of132"))).get_attribute("src"))
  

• 注意：您必须添加以下导入：

  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

• 控制台输出：

  https://www.festo.com/cfp/camosHtml/i?SIG=0020e295a546f45d9acb6844231fd8ff31ca817a_64_64.png
  

在这里你可以找到Ways to deal with #document under iframe的相关讨论

Answer 2

试试这个 更多信息here

 FIREFOX_DRIVER_PATH = "your_geckodriver_path"
 firefox_options = FirefoxOptions()
 firefox_options.headless = True

 # set options as per requirement for firefox
 firefox_options.add_argument("--no-sandbox")
 firefox_options.add_argument("--disable-setuid-sandbox")
 firefox_options.add_argument('--disable-dev-shm-usage')
 firefox_options.add_argument("--window-size=1920,1080")
 driver = webdriver.Firefox(firefox_options=firefox_options, executable_path=FIREFOX_DRIVER_PATH)
 driver.get('http://www.festo.com/cat/it_it/products_VUVG_SCurrentPartNo=8043720')
 time.sleep(5)

 campo_busca = driver.find_elements_by_id('of132')

 print(campo_busca)

Answer 3

从此link下载驱动程序 并将其放置一个文件夹并复制完整路径并粘贴到下面

 FIREFOX_DRIVER_PATH = "driver_path"

 firefox_options = FirefoxOptions()

 #only if you dont want to see the gui else make is false or comment
 firefox_options.headless = True

 driver = webdriver.Firefox(firefox_options=firefox_options, executable_path=FIREFOX_DRIVER_PATH)
 driver.get('http://www.festo.com/cat/it_it/products_VUVG_SCurrentPartNo=8043720')
 time.sleep(3)

 campo_busca = driver.find_elements_by_id('of132')

 print(campo_busca)