抓取遍历多个页面的数据时遇到问题

Question

我用 python 编写了一个脚本来从网页获取数据。该网站在 60 个页面上显示其内容。我的刮刀可以从它的第二页解析数据。当我尝试更改 payload 参数中的页码或创建循环以从少数页面获取数据时，它会立即中断。如何以这种方式纠正我的脚本，以便它可以从所有页面获取数据，而不仅仅是从第二页获取数据。提前致谢。

1. 链接到带有数据的站点：Page_link
2. 链接替换为以下脚本：page_url

我想，页码在这里：

ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$ddlPages:1

这是完整的脚本（仅适用于第 2 页）：

import requests
from bs4 import BeautifulSoup

url = "Link to replace with the above url" ##Replace the number 2 links here

formdata = {
    'searchEntity':'FundServiceProvider',
    'searchType':'Name',
    'searchText':'',
    'registers':'6,29,44,45',
    'AspxAutoDetectCookieSupport':'1'
}
req = requests.get(url,params=formdata,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(req.text,"lxml")

VIEWSTATE = soup.select("#__VIEWSTATE")[0]['value']
EVENTVALIDATION = soup.select("#__EVENTVALIDATION")[0]['value']

payload = {
    '__EVENTTARGET':'','__EVENTARGUMENT':'','__LASTFOCUS':'','__VIEWSTATE':VIEWSTATE,'__SCROLLPOSITIONX':'0','__SCROLLPOSITIONY':'541','__EVENTVALIDATION':EVENTVALIDATION,'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$ddlPages':1,'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$btnNext.x':'260','ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$btnNext.y':'11'
}

with requests.session() as session:
    session.headers = {"User-Agent":"Mozilla/5.0"}
    response = session.post(req.url,data=payload)
    soup = BeautifulSoup(response.text,"lxml")
    tabd = soup.select(".searchresults")[0]
    for items in tabd.select("tr")[:-1]:
        data = ' '.join([item.text for item in items.select("th,td")])
        print(data)

Answer 1

您只需要删除有效载荷数据的最后 2 个字段：

payload = {
    '__EVENTTARGET':'',
    '__EVENTARGUMENT':'',
    '__LASTFOCUS':'',
    '__VIEWSTATE':VIEWSTATE,
    '__SCROLLPOSITIONX':'0',
    '__SCROLLPOSITIONY':'541',
    '__EVENTVALIDATION':EVENTVALIDATION,
    'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$ddlPages':1
}

而不是

payload = {
    '__EVENTTARGET':'',
    '__EVENTARGUMENT':'',
    '__LASTFOCUS':'',
    '__VIEWSTATE':VIEWSTATE,
    '__SCROLLPOSITIONX':'0',
    '__SCROLLPOSITIONY':'541',
    '__EVENTVALIDATION':EVENTVALIDATION,
    'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$ddlPages':1,
    'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$btnNext.x':'260',
    'ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$btnNext.y':'11'
}

然后更新ctl00$cphRegistersMasterPage$gvwSearchResults$ctl18$ddlPages值会得到正确的页面数据