如何在 Python 中使用 BeautifulSoup 查找特定的 HTML 元素

Question

我有一个网站，我正在尝试使用请求和 BeautifulSoup 库从 HTML 中获取特定元素，但我很难找到匹配项。

代码是这样的：

from bs4 import BeautifulSoup
import requests

url = 'https://seekingalpha.com/symbol/AAPL/transcripts'

content = requests.get(url).content
soup = BeautifulSoup(content, "html.parser")

我试图从 Soup 对象中获取此特定标签 <div data-test-id="post-list"> 并尝试过：

soup.find_all('<div data-test-id="post-list">')

但是这会返回一个空列表[]。有人可以帮助我吗？

谢谢

Answer 1

您在页面上看到的数据是从外部来源加载的。 （在源码中，没有<div>s带有data-test-id属性（可以在浏览器中查看——点击Ctrl+U搜索data-test-id）。

要从外部源加载数据，请尝试：

import json
import requests

api_url = "https://seekingalpha.com/api/v3/symbols/aapl/transcripts"
params = {
    "filter[until]": "undefined",
    "id": "aapl",
    "include": "author,primaryTickers,secondaryTickers,sentiments",
    "isMounting": "true",
    "page[size]": "20",
}

data = requests.get(api_url, params=params).json()

# pretty print the data:
# print(json.dumps(data, indent=4))

for d in data["data"]:
    print(
        "{:<50}  {}".format(d["attributes"]["title"][:50], d["links"]["self"])
    )

打印：

Apple Inc. (AAPL) CEO Tim Cook on Q3 2021 Results   /article/4441901-apple-inc-aapl-ceo-tim-cook-on-q3-2021-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q2 2021 Results   /article/4422230-apple-inc-aapl-ceo-tim-cook-on-q2-2021-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q1 2021 Results   /article/4401478-apple-inc-aapl-ceo-tim-cook-on-q1-2021-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q4 2020 Results   /article/4382943-apple-inc-aapl-ceo-tim-cook-on-q4-2020-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q3 2020 Results   /article/4362707-apple-inc-aapl-ceo-tim-cook-on-q3-2020-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q2 2020 Results   /article/4341792-apple-inc-aapl-ceo-tim-cook-on-q2-2020-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q1 2020 Results   /article/4319666-apple-inc-aapl-ceo-tim-cook-on-q1-2020-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q4 2019 Results   /article/4300690-apple-inc-aapl-ceo-tim-cook-on-q4-2019-results-earnings-call-transcript
Apple, Inc. (AAPL) CEO Tim Cook On Q3 2019 Results  /article/4279301-apple-inc-aapl-ceo-tim-cook-on-q3-2019-results-earnings-call-transcript
Apple, Inc. (AAPL) CEO Tim Cook on Q2 2019 Results  /article/4258275-apple-inc-aapl-ceo-tim-cook-on-q2-2019-results-earnings-call-transcript
Apple Inc. (AAPL) CEO Tim Cook on Q1 2019 Results   /article/4236503-apple-inc-aapl-ceo-tim-cook-on-q1-2019-results-earnings-call-transcript
Apple (AAPL) Q4 2018 Results - Earnings Call Trans  /article/4217359-apple-aapl-q4-2018-results-earnings-call-transcript
Apple (AAPL) Q3 2018 Results - Earnings Call Trans  /article/4192790-apple-aapl-q3-2018-results-earnings-call-transcript
Apple (AAPL) Q2 2018 Results - Earnings Call Trans  /article/4168271-apple-aapl-q2-2018-results-earnings-call-transcript
Apple's (AAPL) CEO Tim Cook On Q1 2018 Results - E  /article/4142447-apples-aapl-ceo-tim-cook-on-q1-2018-results-earnings-call-transcript
Apple (AAPL) Q4 2017 Results - Earnings Call Trans  /article/4119877-apple-aapl-q4-2017-results-earnings-call-transcript
Apple (AAPL) Q3 2017 Results - Earnings Call Trans  /article/4093437-apple-aapl-q3-2017-results-earnings-call-transcript
Apple (AAPL) Q2 2017 Results - Earnings Call Trans  /article/4068153-apple-aapl-q2-2017-results-earnings-call-transcript
Apple (AAPL) Q1 2017 Results - Earnings Call Trans  /article/4041266-apple-aapl-q1-2017-results-earnings-call-transcript
Apple (AAPL) Q4 2016 Results - Earnings Call Trans  /article/4014922-apple-aapl-q4-2016-results-earnings-call-transcript