使用Python如何下载所有页面具有相同链接的网页的所有页面的数据答案

【问题标题】：Using Python how to download data of all the pages of a webpage that has the same link for all pages使用Python如何下载所有页面具有相同链接的网页的所有页面的数据
【发布时间】：2021-06-06 17:26:21
【问题描述】：

我正在尝试下载此网页上显示的表格中的所有条目 - https://udhonline.rajasthan.gov.in/Portal/AuctionList 表格中有用于加载下一个条目的按钮，但网页的链接保持不变。我想用 Python 下载所有数据，我尝试了以下方法：

pd.read_html(link)

这会产生一个列表，其中包含表中的前 30 个结果以及另一个包含所有 30 个结果的项目。页面上的默认设置仅显示前 30 个结果。我怎样才能从以下所有页面中获取数据？

【问题讨论】：

标签： python pandas web-scraping

【解决方案1】：

我在页面上进行了枚举

当您点击第 2 页时，您实际上是在向服务器发出 post 请求，请参阅下面的 curl 请求

curl "https://udhonline.rajasthan.gov.in/Portal/SearchAuctionGrid?page=2&Paging=True&pageSize=30&TabViewType=0&UnitId=0" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "X-Requested-With: XMLHttpRequest" -H "Origin: https://udhonline.rajasthan.gov.in" -H "Connection: keep-alive" -H "Referer: https://udhonline.rajasthan.gov.in/Portal/AuctionList" --data-raw "X-Requested-With=XMLHttpRequest"

所以只需向 https://udhonline.rajasthan.gov.in/Portal/SearchAuctionGrid?page=2&Paging=True&pageSize=30&TabViewType=0&UnitId=0

带有这个标题 X-Requested-With=XMLHttpRequest 和这个身体 X-Requested-With=XMLHttpRequest

页面上的项目计数

向该网址发出帖子请求 https://udhonline.rajasthan.gov.in/Portal/SearchAuctionGrid

带有这个标题

X-Requested-With=XMLHttpRequest

还有这些数据

PageSize=50&UnitId=&X-Requested-With=XMLHttpRequest

【讨论】：

【解决方案2】：

您可以使用此示例如何将多个页面中的数据加载到 pandas 数据框中：

import requests
import pandas as pd
from bs4 import BeautifulSoup

api_url = "https://udhonline.rajasthan.gov.in/Portal/SearchAuctionGrid"

params = {
    "page": "1",
    "Paging": "True",
    "pageSize": "30",
    "TabViewType": "0",
    "UnitId": "0",
}

dfs = []
for page in range(1, 4):  # <-- increase number of pages here
    params["page"] = page
    soup = BeautifulSoup(
        requests.post(api_url, params=params).content, "html.parser"
    )
    for t in soup.select("table:not(:has(table))"):
        dfs.append(pd.read_html(str(t))[0].T)

df = pd.concat(dfs).reset_index(drop=True)
print(df)
df.to_csv("data.csv", index=False)

打印：

                                                     0                                                  1                                    2                                             3                                           4                       5                                                  6
0         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar                 Property Number: 220  EMD Deposit Start Date: 01-Jun-2021 08:00 AM  EMD Deposit End Date: 06-Jun-2021 11:59 PM    EMD Deposit Ends In:  Assessed Property Value as per Bid Start Price...
1         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar                 Property Number: 220  EMD Deposit Start Date: 01-Jun-2021 08:00 AM  EMD Deposit End Date: 06-Jun-2021 11:59 PM    EMD Deposit Ends In:  Assessed Property Value as per Bid Start Price...
2         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar   Property Area: 2118.32 Square Feet          Bid Start Date: 03-Jun-2021 10:00 AM          Bid End Date: 07-Jun-2021 11:00 AM            Bid Ends In:  Assessed Property Value as per Bid Start Price...
3         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar   Property Area: 2118.32 Square Feet          Bid Start Date: 03-Jun-2021 10:00 AM          Bid End Date: 07-Jun-2021 11:00 AM            Bid Ends In:  Assessed Property Value as per Bid Start Price...
4         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar              Usage Type: Residential                   EMD Amount (Rs.): 211900.00                    View Details Participate                     NaN  Assessed Property Value as per Bid Start Price...
5         AROGYA NAGAR RESIDENTIAL PLOT NO. 220 [9263]                          Scheme Name: Arogya Nagar              Usage Type: Residential                   EMD Amount (Rs.): 211900.00                    View Details Participate                     NaN  Assessed Property Value as per Bid Start Price...
6          CHANAKYAPURI RESIDENTIAL PLOT NO. 14 [9262]                          Scheme Name: Chanakyapuri                  Property Number: 14  EMD Deposit Start Date: 01-Jun-2021 08:00 AM  EMD Deposit End Date: 06-Jun-2021 11:59 PM    EMD Deposit Ends In:  Assessed Property Value as per Bid Start Price...
7          CHANAKYAPURI RESIDENTIAL PLOT NO. 14 [9262]                          Scheme Name: Chanakyapuri                  Property Number: 14  EMD Deposit Start Date: 01-Jun-2021 08:00 AM  EMD Deposit End Date: 06-Jun-2021 11:59 PM    EMD Deposit Ends In:  Assessed Property Value as per Bid Start Price...

...

并保存 data.csv（来自 LibreOffice 的屏幕截图）：

【讨论】：