如何使用 Beautiful Soup 从复杂的 div 类中抓取内容

Answer 1

这是一个 Selenium 解决方案，用于在 pandas 数据框中获取整个表。

需要进口

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

启动网络驱动程序

# Replace your CHROME DRIVER path here
chrome_path = r"C:\Users\hpoddar\Desktop\Tools\chromedriver_win32\chromedriver.exe"
s = Service(chrome_path)
driver = webdriver.Chrome(service=s)

获取页面

driver.get('https://finance.yahoo.com/quote/BAC/financials?p=BAC')

等待表加载

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//div[@class="D(tbhg)"]')))

获取标题行

headers_elem = driver.find_elements(By.XPATH, '//div[@class="D(tbhg)"]/div/div')
col_headers = [header.text for header in headers_elem]
df = pd.DataFrame(columns = col_headers)
df

Empty DataFrame
Columns: [Breakdown, TTM, 12/30/2021, 12/30/2020, 12/30/2019, 12/30/2018]
Index: []

从表中获取行

这里表中的每一行都存储在rows

rows = driver.find_elements(By.XPATH, '//div[@class="D(tbrg)"]//div[@data-test="fin-row"]')
for row in rows:
    row_values = row.find_elements(By.XPATH, 'div/div')
    df.loc[len(df)] = [row_value.text for row_value in row_values]

输出 ：

这给了我们预期的输出

	Breakdown	TTM	12/30/2021	12/30/2020	12/30/2019	12/30/2018
0	Total Revenue	90,742,000	89,113,000	85,528,000	91,244,000	91,247,000
1	Credit Losses Provision	560,000	4,594,000	-11,320,000	-3,590,000	-3,282,000
2	Non Interest Expense	59,763,000	59,731,000	55,213,000	54,900,000	53,381,000
3	Special Income Charges	-	-	-	-	0
4	Pretax Income	31,539,000	33,976,000	18,995,000	32,754,000	34,584,000
5	Tax Provision	3,521,000	1,998,000	1,101,000	5,324,000	6,437,000
6	Net Income Common Stockholders	26,565,000	30,557,000	16,473,000	25,998,000	26,696,000
7	Diluted NI Available to Com Stockholders	26,565,000	30,557,000	16,473,000	25,998,000	26,696,000
8	Basic EPS	-	3.60	1.88	2.77	2.64
9	Diluted EPS	-	3.57	1.87	2.75	2.61
10	Basic Average Shares	-	8,493,300	8,753,200	9,390,500	10,096,500
11	Diluted Average Shares	-	8,558,400	8,796,900	9,442,900	10,236,900
12	INTEREST_INCOME_AFTER_PROVISION_FOR_LOAN_LOSS	47,080,000	47,528,000	32,040,000	45,301,000	44,150,000
13	Net Income from Continuing & Discontinued Operation	28,018,000	31,978,000	17,894,000	27,430,000	28,147,000
14	Normalized Income	28,018,000	31,978,000	17,894,000	27,430,000	28,147,000
15	Total Money Market Investments	348,000	-90,000	903,000	4,843,000	3,176,000
16	Reconciled Depreciation	1,953,000	1,898,000	1,843,000	1,729,000	2,063,000
17	Net Income from Continuing Operation Net Minority Interest	28,018,000	31,978,000	17,894,000	27,430,000	28,147,000
18	Total Unusual Items Excluding Goodwill	-	-	-	-	0
19	Total Unusual Items	-	-	-	-	0
20	Tax Rate for Calcs	0	0	0	0	0
21	Tax Effect of Unusual Items	0	0	0	0	0

TL:博士

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

chrome_path = r"C:\Users\hpoddar\Desktop\Tools\chromedriver_win32\chromedriver.exe"
s = Service(chrome_path)
driver = webdriver.Chrome(service=s)

driver.get('https://finance.yahoo.com/quote/BAC/financials?p=BAC')

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//div[@class="D(tbhg)"]')))

headers_elem = driver.find_elements(By.XPATH, '//div[@class="D(tbhg)"]/div/div')
col_headers = [header.text for header in headers_elem]
df = pd.DataFrame(columns = col_headers)

rows = driver.find_elements(By.XPATH, '//div[@class="D(tbrg)"]//div[@data-test="fin-row"]')
for row in rows:
    row_values = row.find_elements(By.XPATH, 'div/div')
    df.loc[len(df)] = [row_value.text for row_value in row_values]

结果存储在df.