如何通过单击 xpath 从 url 列表中循环并在 Python 中使用 Selenium 提取数据？

Question

我正在从 URL 列表中提取董事会成员。对于 URL_lst 中的每个 url，单击第一个 xpath（查看更多以展开列表），然后从第二个 xpath（BoardMembers 的信息）中提取值。 以下是我要提取信息的三个公司：https://www.bloomberg.com/quote/FB:US、https://www.bloomberg.com/quote/AAPL:US、https://www.bloomberg.com/quote/MSFT:US

我的代码如下所示，但不起作用。输出列表未聚合。我知道循环有问题，但不知道如何修复它。谁能告诉我如何更正代码？谢谢！

URL_lst = ['https://www.bloomberg.com/quote/FB:US','https://www.bloomberg.com/quote/AAPL:US','https://www.bloomberg.com/quote/MSFT:US']

Outputs = []
driver = webdriver.Chrome(r'xxx\chromedriver.exe')

for url in URL_lst:
    driver.get(url)
    for c in driver.find_elements_by_xpath("//*[@id='root']/div/div/section[3]/div[10]/div[2]/div/span[1]"):
        c.click()
        for e in (c.find_elements_by_xpath('//*[@id="root"]/div/div/section[3]/div[10]/div[1]/div[2]/div/div[2]')[0].text.split('\n'):
            Outputs.append(e)

print(Outputs)

Answer 1

根据您提供的 URL，我为您做了一些重构。我在您尝试单击的每个项目上添加了wait，并通过scrollIntoView Javascript 调用向下滚动到View More 按钮。您最初在循环中单击 View More 按钮，但您的 XPath 仅返回 1 个元素，因此循环是多余的。

我还为董事会成员重构了您的选择器，以直接查询包含其姓名的 div 元素。您的原始查询在实际名称文本上方找到了几个级别的 div，这就是您的 Outputs 列表返回空的原因。

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from time import sleep

URL_lst = ['https://www.bloomberg.com/quote/FB:US','https://www.bloomberg.com/quote/AAPL:US','https://www.bloomberg.com/quote/MSFT:US']

Outputs = []
driver = webdriver.Chrome(r'xxx\chromedriver.exe')

wait = WebDriverWait(driver, 30)

for url in URL_lst:
    driver.get(url)

    # get "Board Members" header
    board_members_header = wait.until(EC.presence_of_element_located((By.XPATH, "//h2[span[text()='Board Members']]")))

    # scroll down to board members
    driver.execute_script("arguments[0].scrollIntoView();", board_members_header)

    # get view more button
    view_more_button = wait.until(EC.presence_of_element_located((By.XPATH, "//section[contains(@class, 'PageMainContent')]/div/div[2]/div/span[span[text()='View More']]")))

    # click view more button
    view_more_button.click()

    # wait on 'View less' to exist, meaning list is expanded now
    wait.until(EC.presence_of_element_located((By.XPATH, "//section[contains(@class, 'PageMainContent')]/div/div[2]/div/span[span[text()='View Less']]")))


    # wait on visibility of board member names
    wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div[contains(@class, 'boardWrap')]//div[contains(@class, 'name')]")))

    # get list of board members names
    board_member_names = driver.find_elements_by_xpath("//div[contains(@class, 'boardWrap')]//div[contains(@class, 'name')]")

    for board_member in board_member_names:
        Outputs.append(board_member.text)

    # explicit sleep to avoid being flagged as bot
    sleep(5)

print(Outputs)

我还在 URL 抓取之间添加了一个明确的 sleep，这样 Bloomberg 就不会将您标记为机器人。