使用从网站中提取的数据创建数据框

Question

我正在尝试使用 selenium 从 Facebook 部分获取共享数量，如下所示：

from selenium import webdriver
import time

chrome_options = webdriver.ChromeOptions()

total = []
shares = []
comments = []
reactions = []
    
query=["www.stackoverflow.com","www.facebook.com"]
driver = webdriver.Chrome(mypath,chrome_options=chrome_options) 
driver.get('https://www.sharedcount.com/')
textarea = driver.find_element_by_xpath('//textarea')
for x in query:
     textarea.send_keys(query)
     button = driver.find_element_by_xpath("//button[@class='button button_accent-green']")
     button.click()
     body = driver.find_element_by_xpath("//tb-data tb-data_face[@class='bold-text']")
     total.append(body)
     sharing=driver.find_element_by_xpath("//tb-line-info__text tb-line-info__text_left tb-line-info__face[@class='bold-text']")
     shares.append(sharing)
     com=driver.find_element_by_xpath("//tb-line-info__text tb-line-info__text_left[@class='bold-text']")
     comments.append(com)

为查询列表中的每个 url 创建一个包含信息的新数据框。 但是，我收到此错误

InvalidSelectorException: Message: invalid selector: Unable to locate an element with the xpath expression //tb-data tb-data_face[@class='bold-text'] because of the following error:
SyntaxError: Failed to execute 'evaluate' on 'Document': The string '//tb-data tb-data_face[@class='bold-text']' is not a valid XPath expression.
  (Session info: chrome=91.0.4472.114)

有谁知道如何修复错误并获得以下输出？

url                       shares         comments         reactions 
www.stackoverflow.com      11.2k           3.8k              9.1k
www.facebook.com         1920.4m           64.9m             517.7m             

使用 WebDriverWait 后，我​​得到了另一个错误：

     24 button.click()
---> 25 WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[@class="tb-data tb-data-face"]/div[@class="bold-text"]')))
     26 body = driver.find_element_by_xpath('//td[@class="tb-data tb-data-face"]/div[@class="bold-text"]')
     27 total.append(body)

~/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/support/wait.py in until(self, method, message)
     78             if time.time() > end_time:
     79                 break
---> 80         raise TimeoutException(message, screen, stacktrace)
     81 
     82     def until_not(self, method, message=''):

TimeoutException: Message: 

Answer 1

我相信您在 XPath 中缺少 <td> 和 <div> 标记。应该是

'//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]'

您还需要使用相同的想法调整sharing 和com XPath。

编辑 1： 尝试等待元素的可见性：

进口：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

for x in query:
     ...
     button.click()
     
     # Wait 10 seconds for the element to be visible
     WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]')))

     body = driver.find_element_by_xpath('//td[@class="tb-data tb-data_face"]/div[@class="bold-text"]')
     total.append(body)
     ...