在网页上应用过滤器时如何复制请求？

Question

我正在尝试在premier league player stats 中应用过滤器时复制请求。我注意到在为 2019/20 赛季应用过滤器时，该 url 添加了组件 '?co=1&se=274'

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274

而不是

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats

但是在做的时候

requests.get('https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274')

并刮掉内容，就像没有应用过滤器一样被刮掉。如何在应用了网页过滤器的地方提出请求？

通过深入研究，我了解到它受 CloudFront 保护，这意味着在发布请求之前，所有查询参数都已被剥离。有没有办法解决这个问题？

这是我抓取数据的方式：

from bs4 import BeautifulSoup as soup
import requests
from tqdm import tqdm
from pprint import pprint
players_url =['https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274']


# this is dict where we store all information:
players = {}
for i in tqdm(players_url):
    player_page = requests.get(i)
    cont = soup(player_page.content, 'lxml')
    time.sleep(2)
    data = dict((k.contents[0].strip(), v.get_text(strip=True)) for k, v in zip(cont.select('.topStat span.stat, .normalStat span.stat'), cont.select('.topStat span.stat > span, .normalStat span.stat > span')))
    clud_ele = cont.find('div', attrs={'class' : 'info'})
    club = {"Club" : clud_ele.get_text(strip=True)}
    position = {"Position": clud_ele.find_next('div', attrs={'class' : 'info'}).get_text(strip=True)}
    data.update(club)
    data.update(position)
    players[cont.select_one('.playerDetails .name').get_text(strip=True)] = data

pprint(players)

在输出中我可以清楚地看到没有应用过滤器，因为本赛季还没有 45 场比赛

{'Virgil van Dijk': {'Accurate long balls': '533',
                     'Aerial battles lost': '207',
                     'Aerial battles won': '589',
                     'Appearances': '122',
                     'Assists': '2',
                     'Big chances created': '11',
                     'Blocked shots': '23',
                     'Clean sheets': '45',

Answer 1

您可以通过复制尝试按季节过滤时完成的后台请求来解决此问题。我使用requests 库来获取所有玩家的统计数据。

这个过程主要涉及三个url，

1. 获取季节ID值。 (ex. 274)

https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100

1. 获取玩家姓名和ID(ex. Name: Virgil van Dijk, ID: 5140)

https://footballapi.pulselive.com/football/players

1. 使用玩家ID(ex. 5140)获取玩家统计数据

https://footballapi.pulselive.com/football/stats/player/

完整的脚本

import requests
import json

response = requests.get('https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100').json() # request to obtain the id values and corresponding season 

id = int(response["content"][0]["id"]) # converts current season id which is a decimal point value to interger

players = {} # dictionary to store players data

playersAndStats = {} # dictionary to store player name and associated stats

numEntries = 100

page = 0

# loop to get player name and id
while True: 

  params = (
      ('pageSize', '100'),
      ('compSeasons', str(id)), 
      ('altIds', 'true'),
      ('page', str(page)),
      ('type', 'player'),
      ('id', '-1'),
      ('compSeasonId', str(id)),
  )

  response = requests.get('https://footballapi.pulselive.com/football/players',params=params).json()

  playersData = response["content"]

  for playerData in playersData:
    players[playerData["id"]] = playerData["name"]["display"]

  page += 1

  if page == response["pageInfo"]["numPages"]:
    break

print("Total no. of players :",len(players))

count = 0 
total = len(players)

# loop to get player stats 
for player in players:

  count += 1
  print(count,"/",total)

  params = (
      ('comps', '1'),
      ('compSeasons', str(id)), # setting season id to current season id
  )

  playerId = str(int(player))

  # gets the stat of the player using playerId 
  response = requests.get('https://footballapi.pulselive.com/football/stats/player/'+playerId,params=params).json()

  stats = response["stats"]

  # creating a stat dict for the player
  playersAndStats[players[player]] = {}

  # loop to store each stat associated with the player
  for stat in stats:
    playersAndStats[players[player]][stat["name"].replace("_"," ").capitalize()] = int(stat["value"])

# to store data to a json file 
f = open("data.json","w")

# pretty prints and writes the same to the json file 
f.write(json.dumps(playersAndStats,indent=4, sort_keys=True))
f.close()

print("Saved to data.json")

样本输出

"Aaron Connolly": {
        "Accurate back zone pass": 1,
        "Accurate fwd zone pass": 1,
        "Accurate pass": 2,
        "Aerial lost": 1,
        "Appearances": 1,
        "Attempts conceded ibox": 2,
        "Attempts conceded obox": 1,
        "Backward pass": 1,
        "Ball recovery": 1,
        "Duel lost": 1,
        "Duel won": 0,
        "Final third entries": 1,
        "Goals conceded": 1,
        "Goals conceded ibox": 1,
        "Losses": 1,
        "Mins played": 24,
        "Open play pass": 1,
        "Poss won mid 3rd": 1,
        "Rightside pass": 1,
        "Successful final third passes": 1,
        "Successful open play pass": 1,
        "Total back zone pass": 1,
        "Total final third passes": 1,
        "Total fwd zone pass": 1,
        "Total offside": 2,
        "Total pass": 2,
        "Total sub on": 1,
        "Total tackle": 0,
        "Touches": 3,
        "Won tackle": 0
    }

data.json 文件包含所有玩家的数据。

与每位球员相关的统计数据会根据他们的比赛位置而有所不同。