使用 BeautifulSoup 进行 Web 抓取 - 无法提取表行

Question

我正在尝试使用 BeautifulSoup 提取以下网页上的表格：

https://www.indiapost.gov.in/VAS/Pages/PMODashboard/DistributionOfPostOffices.aspx

我尝试使用的代码是：

import pandas as pd
from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "https://www.indiapost.gov.in/VAS/Pages/PMODashboard/DistributionOfPostOffices.aspx"
html = urlopen(url)

soup = BeautifulSoup(html, 'lxml')
type(soup)

table = soup.find('table', {'class' : 'tbl'})

#extract rows:

rows = soup.find_all('tr')

最后一行应该打印带有 HTML 标记的行名的输出（即 Sl No.、Head Post Office 等），但它只打印一个空列表。我哪里错了？

Answer 1

您可能希望遵循以下方法来使用请求从该网页获取表格内容。原来您要查找的内容在这个 link 中可用，您可以使用 chrome 开发工具找到它。

工作代码：

import csv
import requests
from bs4 import BeautifulSoup

url = 'https://www.indiapost.gov.in/Documents/DashboardXmlFile/DashboardXML.xml'

def get_tabular_info(link):
    r = requests.get(link)
    soup = BeautifulSoup(r.text,'xml')
    tabular_list = []
    for items in soup.select("DistributionOfPostOffices Table1")[2:]:
        tabular_list.append([item.get_text(strip=True) for item in items.select("A,B,C,D,E,F")])
    return tabular_list

if __name__ == '__main__':
    with open("output_indiapost.csv","w",newline="") as f:
        writer = csv.writer(f)

        for item in get_tabular_info(url):
            writer.writerow(item)
            print(item)

输出如下：

['Sl. No.', 'Circle Name', 'Head Post Office', 'Sub Post Office', 'Branch Post Office', 'Letter Box']
['1', 'Andhra Pradesh Circle', '59', '1535', '8897', '29510']
['2', 'Assam Circle', '19', '606', '3385', '12427']
['3', 'Bihar Circle', '32', '1029', '8031', '22433']
['4', 'Chhattisgarh Circle', '11', '341', '3079', '14988']
['5', 'Delhi Circle', '12', '406', '142', '1187']
['6', 'Gujarat Circle', '33', '1243', '7651', '24377']