Selenium 仅加载页面上找到的一半元素

Question

“我对 selenium 很陌生，我试图在 ebay 的单个页面上获取所有记录，但似乎 webdriver 只加载了所需元素实例的一半，我正在使用 time.sleep 和 webdriver .wait 都加载完整的页面，但当实际页面有 48 个时，它仍然只显示 24 个记录，我也在滚动到页面末尾，所以所有元素都可以加载，但它仍然只显示 24 个实例，这是我的代码。”

{
# creating a random user agent
    ua = UserAgent()
    user_agent = ua.random


# setting up browser
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("user-agent={user_agent}")
    chrome_options.add_argument("start-maximized")
    chrome_options.add_argument('--headless')

# Creating driver

    driver = webdriver.Chrome(executable_path="path to chrome driver", chrome_options=chrome_options)
    driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
    time.sleep(5)

# Scrolling to the end of the page
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 's-item')))

# Looking for required elements
    req_ele = driver.find_elements_by_class_name("s-item")
    for ele in req_ele:
        print(ele)

    driver.quit()
}

“你可以看到实际页面有 48 条记录，但是这段代码只打印出 24 个 selenium 对象”

Answer 1

要打印网页 https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276 上所有 48 个项目的 alt 属性，您需要为 visibility_of_all_elements_located() 诱导 WebDriverWait，您可以使用关注Locator Strategies：

• 使用CSS_SELECTOR：

  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
  print([my_elem.get_attribute("alt") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "section.b-module.b-list.b-listing.srp-list.b-display--landscape img")))])
  

• 使用XPATH：

  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get("https://www.ebay.com/b/Laptops-Netbooks/175672/bn_1648276")
  print([my_elem.get_attribute("alt") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//section[@class='b-module b-list b-listing srp-list b-display--landscape']//img")))])
  

• 控制台输出：

  ['Apple Macbook Pro 15 inch 500 GB Mid 2012 ', 'Google Chrome Pixelbook C0A i7-7Y75 16GB RAM 512GB SSD - Discounted 3444659', 'DELL W21C 3.10GHz 8GB 1TB', 'Dell Latitude E5550 15.6" Intel core i7 5600U 2.6 GHz 8GB 320GB HDD Web cam  ', 'Asus ASUS Vivobook Flip 11.6 HD2-in-1 ConvertibleTouchscreen Laptop, Intel', 'macbook air 2015 13 inch 8gb', 'Apple MacBook Pro 17" Laptop - Working Condition', 'MacBook Pro 13 Inch', 'Apple MacBook Pro A1278\xa0 13.3" Laptop -(April, 2010) Great for School', 'ASUS Computers L402SA Portable Lightweight Laptop PC, Intel Dual Core', 'Dell Vostro 3560 15.5" Laptop Windows 7 Pro 320gb HD', 'MacBook Pro 15" (2016) 500GB SSD 16GB RAM Intel Quad-Core i7 2.7GHz', 'HP Pavilion 15", Used, great condition', '13.3" Fujitsu Lifebook T900 160GB HD Windows 7 Professional', 'Apple MacBook Air 4,2 Core i5-2557M CPU 1.70GHz 4GB 128GB SSD', 'Apple MacBook Pro A1398 Core i7-4980HQ@2.8GHz 16GB 0HD Boots w/GeForce GT 750M', 'ThinkPad X230', 'Apple Macbook Pro 2019 bundle', 'APPLE Macbook Pro 15.4" Retina Core i7 4th 16GB Intel Iris MJLQ2LL/A 2015 READ', 'Sony Vaio Gaming Laptop 2.9GHz Intel i7 12GB RAM Windows 10 Pro 64bit 600GB HDD', 'SAMSUNG CHROMEBOOK PLUS', "Asus G731GU-BI7N9 17.3'' FHD Gaming Laptop i7-9750H 16GB 512GB SSD GTX 1660 Ti", 'Apple MacBook A1278 13.3" Laptop - MB466LL/A (October, 2008)', 'Apple MacBook Pro 15" Laptop with Touchbar and Touch ID, 256GB ', 'MacBook Air 11" 2011', 'HP Pavilion Laptop DV6-2110ea', '2015 MacBook Pro - 13.3" retina display - 8GB RAM- 128GB ', 'Lenovo X1 Carbon ThinkPad i7-4600U 2.69GHz 256GB SSD Pro Dock DU9033S1 Keyboard', "Apple MacBook 12'' 512 GB Gold Laptop - MMGL2LL/A", '??2017 Macbook Pro - 15 Inch W/ Touchbar - Space Gray & Accessories??', 'LENOVO THINKPAD T400 2767-DR8', 'Apple PowerBook G4 15.2" Laptop (10.5) 2005', 'Dell Inspiron 15 7567 15.6" (8GB) - GAMING LAPTOP', 'Sony VAIO VGN-UX280P ', 'Apple MacBook 12" Retina A1534-2746 (2015, Space Gray)', 'Dell Inspiron 15-3521 15.6" Laptop 4GB, Intel processor, WIN 10 ', 'ASUS ROG STRIX GL552VW-DH71 15.6in. (1TB, Intel Core i7 6th Gen., 2.6GHz, 16GB)', 'HP Pavilion x360 11.6 TOUCHSCREEN  Win8 RT3290', 'Apple MacBook Pro 13.3" (128GB SSD, Intel Core i5 8th Gen., 3.90 GHz, 8GB) 2019', 'Acer v3-571-53234g - used - out of box', 'Dell Inspiron 11 3000 Series 4GB RAM - Windows 8', 'Late 2017 MacBook Pro 13 Retina with Apple Care Warranty- Pristine Condition', 'IBM Thinkpad 390e Laptop Intel Celetron 300Mhz SEE NOTES', 'Dell Precision 5510 P56F Laptop Intel Core i7-6820HQ 2.7Ghz 8GB SEE NOTES', 'Sony Vaio P90HS P Series UMPC Intel Z540 1.87GHz 60GB HDD 2GB RAM WINDOWS 10', 'one laptop per child (OLTP)', 'HP Spectre x360 15.6" 4K OLED - Intel Core i7 - 16GB RAM - 1TB SSD (15-df0070nr)', 'HP Pavilion DV 6500 Laptop']
  

Answer 2

问题在你等待：

WebDriverWait(driver, 30).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, 's-item')))

您不是在等待新元素，而是在等待具有该类名称的所有元素都可见。所以如果新的 24 还没有被附加到 dom 上，等待无关紧要。

您需要等待计数超过 24 并且它们可见。 几个潜在的解决方案： https://sqa.stackexchange.com/questions/7018/how-to-wait-until-the-count-of-a-web-element-changes-in-webdriver 要么 https://huddle.eurostarsoftwaretesting.com/how-to-selenium-expected-conditions/