如何使用 BeautifulSoup 排除表中的某些行？

Question

代码工作正常，但是，我尝试获取表格的 URL 似乎在整个表格中重复了标题，我不确定如何处理这个问题并在我尝试删除这些行时将数据导入 BigQuery，某些字符是不允许的。

URL = 'https://www.basketball-reference.com/leagues/NBA_2020_games-august.html'

driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get(URL)
soup = BeautifulSoup(driver.page_source,'html')
driver.quit()
tables = soup.find_all('table',{"id":["schedule"]})

table = tables[0]
tab_data = [[cell.text for cell in row.find_all(["th","td"])]
                        for row in table.find_all("tr")]
json_string = ''
headers = [col.replace('.', '_').replace('/', '_').replace('%', 'pct').replace('3', '_3').replace('(', '_').replace(')', '_') for col in tab_data[1]]
for row in tab_data[2:]:
    json_string += json.dumps(dict(zip(headers, row))) + '\n'
with open('example.json', 'w') as f:
    f.write(json_string)

    print(json_string)

Answer 1

您可以将tr 行中的class 设为无，这样您就不会得到重复的标题。

以下代码从表中创建一个数据框

from bs4 import BeautifulSoup
import requests
import pandas as pd

res = requests.get("https://www.basketball-reference.com/leagues/NBA_2020_games-august.html")

soup = BeautifulSoup(res.text, "html.parser")
table = soup.find("div", {"id":"div_schedule"}).find("table")
columns = [i.get_text() for i in table.find("thead").find_all('th')]

data = []

for tr in table.find('tbody').find_all('tr', class_=False):
    temp = [tr.find('th').get_text(strip=True)]
    temp.extend([i.get_text(strip=True) for i in tr.find_all("td")])
    data.append(temp)

df = pd.DataFrame(data, columns = columns)

print(df)

输出：

                Date Start (ET)         Visitor/Neutral  PTS           Home/Neutral  PTS               Attend. Notes
0    Sat, Aug 1, 2020      1:00p              Miami Heat  125         Denver Nuggets  105  Box Score
1    Sat, Aug 1, 2020      3:30p               Utah Jazz   94  Oklahoma City Thunder  110  Box Score
2    Sat, Aug 1, 2020      6:00p    New Orleans Pelicans  103   Los Angeles Clippers  126  Box Score
3    Sat, Aug 1, 2020      7:00p      Philadelphia 76ers  121         Indiana Pacers  127  Box Score
4    Sat, Aug 1, 2020      8:30p      Los Angeles Lakers   92        Toronto Raptors  107  Box Score
..                ...        ...                     ...  ...                    ...  ...        ... ..     ...   ...
75  Thu, Aug 13, 2020             Portland Trail Blazers               Brooklyn Nets
76  Fri, Aug 14, 2020                 Philadelphia 76ers             Houston Rockets
77  Fri, Aug 14, 2020                         Miami Heat              Indiana Pacers
78  Fri, Aug 14, 2020              Oklahoma City Thunder        Los Angeles Clippers
79  Fri, Aug 14, 2020                     Denver Nuggets             Toronto Raptors

[80 rows x 10 columns]

要插入到 bigquery，您可以使用 https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_gbq.html 直接将 json 插入到 bigquery 或将数据帧插入到 bigquery