有没有一种方法可以将所有内容都具有相同名称的站点进行 webscrapp？

Question

Hi ! I'm new to Beautifulsoup, I was trying to webscrapp the info from this website:

问题是，当我尝试检查网站上的元素时，所有内容都称为“td”和类“sch1”。因此，当我尝试导入时，我变得一团糟。我怎样才能以一种可读和可用的方式导入这些信息，也许我会尝试用它构建一个数据框。

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://feeds.donbest.com/schedulemembers/getRotation.html?bookType=1&eventDate=20230129"
get_url = requests.get(url).content
soup = BeautifulSoup(get_url,"html.parser")

title = soup.find_all("td","schtop1")
 rotation = soup.find_all("td","sch1")

 title_list = []
 rotation_list = []

 for mainT in title:
     title_list.append(mainT.text)
 print(title_list)

 for rot in rotation:
     rotation_list.append(rot.text)
print(rotation_list)

输出： ['NFL 会议锦标赛'，'2023 年 1 月 29 日，星期日'] ['321', '旧金山 49ERS', '', 'P: Sun Jan 29 12:00:00 PST 2023\xa0\n C: Sun Jan 29 14:00:00 PST 2023\xa0\n E: Sun 2023 年 1 月 29 日 15:00:00 PST', '322', '费城老鹰队', '323', '辛辛那提猛虎队', '', 'P: Sun Jan 29 15:30:00 PST 2023\xa0\n C : Sun Jan 29 17:30:00 PST 2023\xa0\n E: Sun Jan 29 18:30:00 PST 2023', '324', 'KANSAS CITY CHIEFS']

我需要能够使用这些信息来构建一个如下所示的 pandas 数据框：

Date	Rot Visitor	Visitor	Rot Home	Home	PST	ET	CT
SUNDAY, JANUARY 29, 2023	321	SAN FRANCISCO 49ERS	322	PHILADELPHIA EAGLES	Sun Jan 29 12:00:00 PST 2023	Sun Jan 29 15:00:00 PST	C: Sun Jan 29 14:00:00 PST 2023
SUNDAY, JANUARY 29, 2023	323	PHILADELPHIA EAGLES	324	CINCINNATI BENGALS	Sun Jan 29 15:30:00 PST	Sun Jan 29 18:30:00 PST 2023	Sun Jan 29 17:30:00 PST 2023

如果我能以更有用的格式获取数据，我认为我可以构建数据框。

Answer 1

import pandas as pd


df = pd.read_html(
    'https://feeds.donbest.com/schedulemembers/getRotation.html?bookType=1&eventDate=20230129/')[0]
print(df)

输出：

0                       NFL CONFERENCE CHAMPIONSHIPS  ...  NFL CONFERENCE CHAMPIONSHIPS
1                           SUNDAY, JANUARY 29, 2023  ...      SUNDAY, JANUARY 29, 2023
2  321  SAN FRANCISCO 49ERS  P: Sun Jan 29 12:00:...  ...                           NaN
3  323  CINCINNATI BENGALS  P: Sun Jan 29 15:30:0...  ...                           NaN

[4 rows x 7 columns]

Answer 2

尝试：

import re
import pandas as pd
import requests
from bs4 import BeautifulSoup


url = 'https://feeds.donbest.com/schedulemembers/getRotation.html?bookType=1&eventDate=20230129/'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')


all_data = []
for t in soup.select('table:not(:has(table))'):
    rows = []
    for tr in t.select('tr'):
        tds = [td.text for td in tr.select('td')]
        rows.append(tds)
    all_data.append({
        'Date': soup.select('td[colspan="7"]')[1].text.strip(),
        'Rot Visitor': rows[0][0],
        'Visitor': rows[0][1],
        'Rot Home': rows[2][0],
        'Home': rows[2][1],
        'Dates': {k.strip(): v.strip() for k, v in re.findall(r'(?sm)(S+)s*:(.*?)(?:[PEC]:|$)', rows[1][1])}
    })

df = pd.DataFrame(all_data)
df = pd.concat([df, df.pop('Dates').apply(pd.Series)], axis=1)
df = df.rename(columns={'P': 'PST', 'E': 'ET', 'C': 'CT'})
print(df.to_markdown())

印刷：

	Date	Rot Visitor	Visitor	Rot Home	Home	PST	CT	ET
0	SUNDAY, JANUARY 29, 2023	321	SAN FRANCISCO 49ERS	322	PHILADELPHIA EAGLES	Sun Jan 29 12:00:00 PST 2023	Sun Jan 29 14:00:00 PST 2023	Sun Jan 29 15:00:00 PST 2023
1	SUNDAY, JANUARY 29, 2023	323	CINCINNATI BENGALS	324	KANSAS CITY CHIEFS	Sun Jan 29 15:30:00 PST 2023	Sun Jan 29 17:30:00 PST 2023	Sun Jan 29 18:30:00 PST 2023