如何使用 Beautiful Soup 和 Pandas 在多个网页中抓取表格？

Question

我是 python 和 bs4 的新手。我已经尝试了几个小时使用漂亮的汤和熊猫在几个网页中刮一张桌子。当我抓取 2 个页面时它起作用了，但是当我尝试抓取所有 13 个网页时，我遇到了麻烦。当我将 range 函数从 2 更改为 13 时，代码不会生成 DF 或 CSV 文件。我做错了什么？

dfs=[]

for page in range(13):
    http = "http://websitexample/Records?year=2020&page={}".format(page+1)

    url = requests.get(http)
    soup = BeautifulSoup(url.text, "lxml")
    table = soup.find('table')
    df_list = pd.read_html(url.text)
    df = pd.concat(df_list)
    
    links = []
    for tr in table.find_all("tr"): 
        trs = tr.find_all("td")
        for each in trs:
            try: 
                link = each.find('a')['href']
                links.append(link)
            except: 
                pass
    
    df['Link']= links
    dfs.append(df)

final_df = pd.concat(dfs)
final_df.to_csv("NewFileAll13.csv",index=False,encoding='utf-8-sig')

我收到错误消息：

值错误：值的长度与索引的长度不匹配。

我非常感谢您提供的任何建议。谢谢！

Answer 1

要从所有页面下载所有数据+链接，您可以使用此示例：

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = 'http://reactwarn.floridajobs.org/WarnList/Records'
params = {
    'year': 2020,
    'page': 1
}

all_data = []
for params['page'] in range(1, 14):
    print('Page {}..'.format(params['page']))

    soup = BeautifulSoup(requests.get(url, params=params).content, 'lxml')

    for row in soup.select('tbody tr'):
        tds = [td.get_text(strip=True, separator='\n') for td in row.select('td')][:-1] + [row.a['href'] if row.a else '']
        all_data.append(tds)

df = pd.DataFrame(all_data, columns=['Company Name', 'State Notification Date', 'Layoff Date', 'Employees Affected', 'Industry', 'Attachment'])
print(df)
df.to_csv('data.csv', index=False)

打印：

                                           Company Name  ...                                         Attachment
0     TrueCore Behavioral Solutions\n5050 N.E. 168th...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
1     Cuba Libre Orlando, LLC t/a Cuba Libre Restaur...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
2     Hyatt Regency Orlando\n9801 International Dr.O...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
3     ABM. Inc.\nNova Southeastern University3301 Co...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
4     Newport Beachside Resort\n16701 Collins Avenue...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
...                                                 ...  ...                                                ...
1251  P.F. Chang's China Bistro\n3597 S.W. 32nd Ct.,...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
1252  P.F. Chang's China Bistro\n11361 N.W. 12th St....  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
1253  P.F. Chang's China Bistro\n8888 S.W. 136th St....  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
1254  P.F. Chang's China Bistro\n17455 Biscayne Blvd...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...
1255  Grand Hyatt Tampa Bay\n2900 Bayport DriveTAMPA...  ...  /WarnList/Download?file=%5C%5Cdeo-wpdb005%5CRe...

[1256 rows x 6 columns]

并保存data.csv（来自 LibreOffice 的屏幕截图）：