【发布时间】：2020-10-10 15:43:12
【问题描述】：
我正在尝试抓取 Naukri 的招聘信息。网络抓取太耗时，所以我切换到网络请求。我相信我通过正确更改 URL（而不是单击下一个选项卡）获得了分页请求模式。
网址示例：
https://www.naukri.com/maintenance-jobs?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-2?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-3?xt=catsrch&qf%5B%5D=19
https://www.naukri.com/maintenance-jobs-4?xt=catsrch&qf%5B%5D=19
下面的代码在第一页上正常工作。但是，它无法从下一个请求链接进行分页和获取数据。相反，对于每次运行，它都从第一个链接获取数据。
我哪里出错了？如何使用 URL 请求正确分页？
import pandas as pd
import time
import re
from bs4 import BeautifulSoup
import unicodecsv
import requests
import ast
import json
import time
import logging
import datetime
headers_naukri={
"authority": 'www.naukri.com',
"user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Edg/85.0.564.70',
'x-newrelic-id': 'VgUHWVFXABACXVdaBAgHUF0=',
"systemid": '109',
"content-type": "application/json",
"accept": 'application/json',
"cache-control": "no-cache",
"clientid": 'd3skt0p',
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
'referer': 'https://www.naukri.com/maintenance-jobs?xt=catsrch&qf[]=19',
"sec-fetch-site": "same-origin",
"accept-encoding": "gzip, deflate, br",
"accept-language": "en-US,en;q=0.9",
"appid": '109',
"content-type": "application/json"
}
session = requests.Session()
source_url='https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong='
print(source_url)
try:
req = session.get(source_url, headers = headers_naukri)
except requests.exceptions.TooManyRedirects:
time.sleep(15)
req = session.get(source_url,headers=headers_naukri)
source_name = 'naukri'
source_id = '1'
source = req.text
#print(source)
source_json = json.loads(source)
noOfJobs=source_json['noOfJobs']
print(noOfJobs)
pages = (noOfJobs // 20) + 1
print(pages)
print(source_json['jobDetails'][0]['jobId'])
a=headers_naukri['referer'].split("?")
for i in range(2,pages,1):
nexturl=source_url.replace('&src=', '-'+str(i)+'&src=')
nexturl=nexturl.replace('&xt', '&pageNo='+str(i)+'&xt')
headers_naukri['referer']= a[0]+'-'+str(i)+'?'+a[1]
print(headers_naukri['referer'])
#print(nexturl)
try:
req = session.get(source_url, headers = headers_naukri)
headers_naukri['referer']= headers_naukri['referer'][:-18]+'-'+str(i)+headers_naukri['referer'][-18:]
except requests.exceptions.TooManyRedirects:
time.sleep(15)
req = session.get(source_url,headers=headers_naukri)
source_name = 'naukri'
source_id = '1'
source = req.text
#print(source)
source_json = json.loads(source)
print(source_json['jobDetails'][1]['jobId'])
输出：
https://www.naukri.com/jobapi/v3/search?noOfResults=20&urlType=search_by_keyword&searchType=adv&keyword=maintenance&xt=catsrch&functionAreaId=19&seoKey=maintenance-jobs&src=jobsearchDesk&latLong=
143263
7164
300920001858
https://www.naukri.com/maintenance-jobs-2?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-3?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-4?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-5?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-6?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-7?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-8?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-9?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-10?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-11?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-12?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-13?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-14?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-15?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-16?xt=catsrch&qf[]=19
300920001858
https://www.naukri.com/maintenance-jobs-17?xt=catsrch&qf[]=19
可以清楚的看到，在每个请求上，第一个job post id都是一样的，并且没有分页。
【问题讨论】：
标签： python pandas web-scraping beautifulsoup python-requests