python中的Selenium网络抓取无法读取元素的.text

Question

我正在尝试从 verizon 网站上删除评论，我通过在网页上执行 inspect 找到了评论的 xpath。我正在执行下面的代码，但是这个review.text 似乎并没有一直完美地工作。我有时会收到正确的文字，有时它只是打印Error in 消息 -

不确定，我做错了什么..

from selenium import webdriver

url = 'https://www.verizonwireless.com/smartphones/samsung-galaxy-s7/'
browser = webdriver.Chrome(executable_path='/Users/userName/PycharmProjects/Verizon/chromedriver')
browser.get(url)
reviews = []
xp = '//*[@id="BVRRContainer"]/div/div/div/div/div[3]/div/ul/li[2]/a/span[2]'

# read first ten pages of reviews ==> 
for j in range(10):
    reviews.extend(browser.find_elements_by_xpath('//*[@id="BVRRContainer"]/div/div/div/div/ol/li[*]/div/div[1]'
                                                  '/div/div[2]/div/div/div[1]/p'))
    try:
        next = browser.find_element_by_xpath(xp)
        next.click()
    except:
        print(j,"error clicking")

# Print reviews  ===>
for i, review in enumerate(reviews):
    try:
         print(review.text)
    except:
         print("Error in :" review)

Answer 1

您应该改进代码的逻辑。请注意，在重定向到下一页后，您无法从第一页获取元素的文本 - 您需要在单击“下一步”按钮之前获取文本。

尝试使用以下代码：

from selenium import webdriver
from selenium.common.exceptions import WebDriverException
import time

url = 'https://www.verizonwireless.com/smartphones/samsung-galaxy-s7/'
browser = webdriver.Chrome()
browser.get(url)
reviews = []
xp = '//a[span[@class="bv-content-btn-pages-next"]]'

# read first ten pages of reviews ==> 
for i in range(10):
    for review in browser.find_elements_by_xpath('//div[@class="bv-content-summary-body-text"]/p'):
        reviews.append(review.text)
    try:
        next = browser.find_element_by_xpath(xp)
        next.location_once_scrolled_into_view
        time.sleep(0.5) # To wait until scrolled down to "Next" button
        next.click()
        time.sleep(2) # To wait for page "autoscrolling" to first review + until modal window dissapeared
    except WebDriverException:
        print("error clicking")


for review in reviews:
    print(review)