如何单击按钮以使用 python 抓取评论

Question

我正在尝试抓取此网站上的评论：https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/。

但是，我必须单击一个按钮 (Toon meer) 才能显示所有评论。

<div class="load-more load-more--divider load-more--reviews js-review-load-more-container">
  <a data-href="/nl/rnwy/productPage/reviews?productId=9200000118425896&amp;offset=5&amp;limit=10&amp;loadMore=true" class="review-load-more__button js-review-load-more-button" data-test="review-load-more"><div class="css-loader css-loader--reviews"></div>
      Toon meer</a>
</div>

我使用下面的代码：

import requests
import pandas as pd
from selenium import webdriver
from bs4 import BeautifulSoup
from datetime import datetime

start_time = datetime.now()
data = []
link = "https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425897/"
op = webdriver.ChromeOptions()
op.add_argument('--ignore-certificate-errors')
op.add_argument('--incognito')
op.add_argument('--headless')
driver = webdriver.Chrome(executable_path='D:/Desktop/work/real/chromedriver.exe',options=op)
driver.get(link)
driver.find_element_by_css_selector('div.review-load-more__button js-review-load-more-button').click()

但是，它会引发错误：

No such element: Unable to locate element: {"method":"css selector","selector":"div.review-load-more__button js-review-load-more-button"} . 

有什么解决办法吗？

Answer 1

Css 选择器不能通过包含文本来选择元素。

尝试使用 xpath。脚本的最后一行应该类似于：

wait = WebDriverWait(driver, 10)
wait.until(expected_conditions.element_to_be_clickable((By.XPATH, "//a[contains(., 'Toon meer')]")).click()

Answer 2

当您获得页面时，会弹出一个带有接受按钮的弹出窗口，单击它，然后继续单击您的元素。

driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[@class='js-confirm-button']"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[@data-test='review-load-more']"))).click()

导入

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

Answer 3

要点击Toon meer，你需要为element_to_be_clickable()诱导WebDriverWait，你可以使用以下Locator Strategies之一：

• 使用CSS_SELECTOR：

  driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-test='consent-modal-confirm-btn']>span"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.review-load-more__button.js-review-load-more-button"))).click()
  

• 使用XPATH：

  driver.get('https://www.bol.com/nl/p/Matras-140x200-7-zones-koudschuim-premium-plus-tijk-15-cm-medium/9200000118425896/')
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@data-test='consent-modal-confirm-btn']/span"))).click()
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@class='review-load-more__button js-review-load-more-button' and contains(., 'Toon meer')]"))).click()
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC