【发布时间】：2021-10-06 23:30:35
【问题描述】：
伙计们，我正在尝试使用 selenium 抓取亚马逊评论，但不知道如何处理下一页 URL 我想使用动态条件进行抓取，而不是通过自计数页面并应用静态方法
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
#Using chrome browser
driver=webdriver.Chrome(executable_path='./chromedriver.exe')
driver.get('https://www.amazon.in/Skybags-Brat-Black-Casual-Backpack/dp/B08Z1HHHTD/ref=sr_1_2?dchild=1&keywords=skybags&qid=1627786382&sr=8-2')
title_of_product = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "productTitle"))
)
print(title_of_product.text)
Reviews=WebDriverWait(driver, 10).until(
EC.presence_of_all_elements_located((By.XPATH, "//span[@class='a-size-base review-text review-text-content']/span")))
next_button =WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME,"a-last"))).click()
time.sleep(10)
driver.close()
【问题讨论】：
-
看看代码工作正常，（处理下一页 URL）意味着我想通过单击下一步导航到此链接，现在工作正常我想知道如何使用 for 循环或处理它们多页的东西并获得评论amazon.in/Skybags-Brat-Black-Casual-Backpack/product-reviews/…
标签： python selenium web-scraping pagination selenium-chromedriver