通过控制 url 上的 pageNo 来循环浏览页面

Question

对 python 非常陌生，需要有关如何通过增加我需要抓取的 url 上的 pageNo 值正确循环链接的帮助，然后不断将结果附加到 df

以下是我所拥有的。

链接在登录墙后面，但希望您不需要它来查看我的代码。

提前谢谢你！

from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import urllib.request
import pandas as pd
import time
from pymongo import MongoClient

#database
client = MongoClient("xxx")
db = client["xxx"]
collection = db["xxx"]

# launch WMS
url = "https://xxx.inserdomain.com/solution/login.htm"
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)

#Login start
username = driver.find_element_by_id("username")
username.clear()
username.send_keys("xxx")

password = driver.find_element_by_id("password")
password.clear()
password.send_keys("xxx")

driver.find_element_by_id("loginButton").click()
#login end

# open shipments page
pageSize = 200
pageNo = 0
currentView = 34448
url2 = "https://xxx.insertdomain.com/solution/entitylist.htm?entityName=Shipment&tabName=Shipment&pageNo={pageNo}&pageSize={pageSize}&currentViewId={currentView}".format(pageNo=pageNo, pageSize=pageSize, currentView=currentView)
driver.get(url2)
html = driver.page_source

dfs = pd.read_html(html, attrs={"class":"roundedTable"}, header=5)

for df in dfs:
    df.dropna(how="all", axis="columns", inplace=True),
    df.drop("No", axis="columns", inplace=True),
    df.dropna(how='all', axis=0, inplace=True),
    print(df.to_json(orient="index"))

df.reset_index(inplace=True)
df_dict = df.to_dict("records")
collection.insert_many(df_dict)

Answer 1

下面的示例循环了 100 页（包括 0 - 99）。您可以使用 break 退出循环。

for pageNo in range(100): # replace 100 with the maximum page number  
    url2 = "https://rby.deposco.com/solution/entitylist.htm?entityName=Shipment&tabName=Shipment&pageNo={pageNo}&pageSize={pageSize}&currentViewId={currentView}".format(pageNo=pageNo, pageSize=pageSize, currentView=currentView)
    ...