使用 BeautifulSoup 从 UPS 网站提取表格

Question

我试图使用下面的代码通过表格 ID 提取我们的柴油附加费，但直到 <thead> 而不是 <tbody> 之前它的唯一读数是我做错了什么？

url = 'https://www.ups.com/us/en/shipping/surcharges/fuel-surcharges.page'
response = requests.get(url)
print(response.status_code)
soup = BeautifulSoup(response.content,"html.parser")
tables = soup.find(id="USDiesel")
print(tables)
tables_all = []
for tr in tables.find_all('tr'):
    data = []
    for td in tr.find_all('td'):
        data.append(td.text.strip())
    tables_all.append(data)
table_df = pd.DataFrame(tables_all)
headers = table_df.iloc[0]
UPS_Gfuelsurcharge_df  = pd.DataFrame(table_df.values[1:], columns=headers)
print(UPS_Gfuelsurcharge_df)

Answer 1

网站使用ajax加载表格数据。

import requests

res = requests.get("https://www.ups.com/assets/resources/fuel-surcharge/us.json")

print(res.json())