如何从 h2 标签中获取 HREF？蟒蛇/硒

Question

网址：https://www.nielsen.com/us/en/insights/related-tag/covid-19/

我有以下 HTML 元素：

我想从这个类中提取href和标题文本。

我使用了driver.find_element_by_class_name('h2.entry-title h1>a').get_attribute('href)，但没有返回。

有什么建议吗？

Answer 1

你一开始做的就是错的。您可能想使用driver.find_element_by_css_selector() 而不是driver.find_element_by_class_name()。但是，更好的方法是使用请求库，因为您要查找的内容在页面源中可用。尝试运行以下脚本，让它们都遍历多个页面。

import requests
from bs4 import BeautifulSoup

link = 'https://www.nielsen.com/us/en/insights/related-tag/covid-19/'

while True:
    r = requests.get(link)
    soup = BeautifulSoup(r.text,"html.parser")
    for item in soup.select("article"):
        article_link = item.select_one("h2.entry-title > a[href]")['href']
        article_title = item.select_one("h2.entry-title > a[href]").get_text(strip=True)
        print(article_link,article_title)

    try:
        link = soup.select_one("a[class^='next']")['href']
    except TypeError:
        break

如果您仍然想坚持使用硒，您可以这样做：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

with webdriver.Chrome() as driver:
    driver.get("https://www.nielsen.com/us/en/insights/related-tag/covid-19/")

    while True:
        for elem in WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"article"))):
            article_link = WebDriverWait(elem,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"h2.entry-title > a[href]"))).get_attribute("href")
            article_title = WebDriverWait(elem,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"h2.entry-title > a[href]"))).text
            print(article_link,article_title)

        try:
            WebDriverWait(driver,5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[class^='next']"))).click()
            WebDriverWait(driver,5).until(EC.staleness_of(elem))
        except:
            break