如何抓取ajax返回的网页内容？

Question

这是我要抓取的页面：https://www.racing.com/form/2018-11-06/flemington/race/7/results 比赛结果信息不在源代码中。

我在 Chrome DevTools 中尝试过，但没有找到包含结果的响应数据。

以下是源代码中的一些代码：

ng-controller="formTabResultsController" 
ng-init="meet=5149117;race=7;init();" ajax-loader="result"

我认为结果被返回并保存在“结果”结构中，因为有很多这样的：“result.PrizeMoney”“result.Record”。

那么如何用 Python 获取结果的数据呢？谢谢。

Answer 1

此站点在https://graphql.rmdprod.racing.com 上使用GraphQL API。 API 密钥需要通过标头发送并检索到here。

curl、sed 和 jq 的示例：

api_key=$(curl -s "https://www.racing.com/layouts/app.aspx" | \
          sed -nE 's/.*headerAPIKey:\s*"(.*)"/\1/p')

curl -s "https://www.racing.com/layouts/app.aspx"
query='query GetMeeting($meetCode: ID!) {
  getMeeting(id: $meetCode) {
    id
    trackName
    date
    railPosition
    races {
      id
      raceNumber
      status
      tempo
      formRaceEntries {
        id
        raceEntryNumber
        horseName
        silkUrl
        jockeyName
        trainerName
        scratched
        speedValue
        barrierNumber
        horse {
            name
            fullName
            colour
        }
      }
    }
  }
}'
variables='{ "meetCode": 5149117 }'
curl -G 'https://graphql.rmdprod.racing.com' \
     --data-urlencode "query=$query" \
     --data-urlencode "variables=$variables" \
     -H "X-Api-Key: $api_key" | jq '.'

将python 与python-requests 一起使用：

import requests
import re
import json

r = requests.get("https://www.racing.com/layouts/app.aspx")
api_key = re.search(".*headerAPIKey:\s*\"(.*)\"", r.text).group(1)

query= """query GetMeeting($meetCode: ID!) {
  getMeeting(id: $meetCode) {
    id
    trackName
    date
    railPosition
    races {
      id
      raceNumber
      status
      tempo
      formRaceEntries {
        id
        raceEntryNumber
        horseName
        silkUrl
        jockeyName
        trainerName
        scratched
        speedValue
        barrierNumber
        horse {
            name
            fullName
            colour
        }
      }
    }
  }
}"""
payload = {
    "variables": json.dumps({ 
        "meetCode": 5149117 
    }), 
    "query": query
}
r = requests.get(
    'https://graphql.rmdprod.racing.com', 
    params = payload,
    headers = {
        "X-Api-Key": api_key
    })
print(r.json())

Answer 2

Chrome 开发工具显示对其 API 的调用

import re
import requests
import json


resp = requests.get('https://api.racing.com/v1/en-au/race/results/5149117/7/?callback=angular.callbacks._b')

# Returned JSONP so we remove the function call: keep only what is between ()
m = re.search(r'\((.*)\)', resp.text, flags=re.S)
data = json.loads(m.group(1))

print(data.keys())
#    dict_keys(['race', 'resultCollection', 'exoticCollection'])

print(data['resultCollection'][0])
# {'position': {'at400m': 12, 'at800m': 20, 'finish': 1, 'positionAbbreviation': '1st', 'positionDescription': '', 'positionType': 'Finished'}, 'scratched': False, 'winningTime': 20117, 'margin': None, 'raceEntryNumber': 23, 'number': 23, 'barrierNumber': 19, 'isDeadHeat': False, 'weight': '51kg', 'rating': {'handicapRating': 109, 'ratingProgression': 0}, 'prizeMoney': 4000000.0, 'horse': {'fullName': 'Cross Counter (GB)', 'code': 5256710, 'urlSegment': 'cross-counter-gb', 'silkUrl': '//s3-ap-southeast-2.amazonaws.com/racevic.silks/bb/12621.png', 'age': 5, 'sex': 'Gelding', 'colour': 'Bay', 'sire': 'Teofilo (IRE)', 'dam': 'Waitress (USA)', 'totalPrizeMoney': '$4,576,227', 'averagePrize': '$508,470'}, 'trainer': {'fullName': None, 'shortName': 'C.Appleby', 'code': 20658431, 'urlSegment': 'charlie-appleby-gb'}, 'jockey': {'fullName': 'K.McEvoy', 'shortName': 'K.McEvoy', 'code': 25602, 'urlSegment': 'kerrin-mcevoy', 'allowedClaim': 0.0, 'apprentice': False}, 'gear': {'hasChanges': True, 'gearCollection': [{'changeDate': '2018-11-02T00:00:00', 'currentChange': True, 'description': 'Bandages (Front): On', 'name': 'Bandages (Front)', 'status': 'On', 'comments': None}, {'changeDate': '2018-08-01T00:00:00', 'currentChange': False, 'description': 'Ear Muffs (Pre-Race Only)', 'name': 'Ear Muffs (Pre-Race Only)', 'status': 'On', 'comments': None}, {'changeDate': '2018-08-01T00:00:00', 'currentChange': False, 'description': 'Lugging Bit', 'name': 'Lugging Bit', 'status': 'On', 'comments': 'Rubber ring bit'}, {'changeDate': '2018-08-01T00:00:00', 'currentChange': False, 'description': 'Cross-over Nose Band', 'name': 'Cross-over Nose Band', 'status': 'On', 'comments': None}], 'currentGearCollection': None}, 'odds': {'priceStart': '$9.00', 'parimutuel': {'returnWin': '12', 'returnPlace': '4.40', 'isFavouriteWin': False}, 'fluctuations': {'priceOpening': '$10.00', 'priceFluc': '$10.00'}}, 'comment': 'Bit Slow Out Settled Down near tail lucky to avoid injured horse was checked though 12l bolting Turn Straightened Up Off Mid-Field 7-8l gets Clear 400 and charged home to score. big win # very good from back', 'extendedApiUrl': '/v1/en-au/form/horsestat/5149117/7/5256710', 'extendedApiUrlMobile': '/v1/en-au/form/horsestatmobile/5149117/7/5256710', 'last5': ['-', '4', '3', '-', '4']}

Answer 3

另一种方法是使用这些参数（可通过浏览器中的“开发人员”选项卡发现），而不使用正则表达式：

import requests
import json

url = 'https://graphql.rmdprod.racing.com/?query=query%20GetMeeting($meetCode:%20ID!)%20%7BgetMeeting(id:%20$meetCode)%7Bid,trackName,date,railPosition,races%7Bid,raceNumber,status,tempo,formRaceEntries%7Bid,raceEntryNumber,horseName,silkUrl,jockeyName,trainerName,scratched,speedValue,barrierNumber%7D%7D%7D%7D&variables=%7B%20%22meetCode%22:%205149117%20%7D'


headers =     {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0",
        "Accept": "*/*",
        "Accept-Language": "en-US,en;q=0.5",
        "Content-Type": "application/json",
        "X-Api-Key": "da2-akkuiub3brhahc7nab2msruddq"
    }


resp = requests.get(url,headers=headers)
data= json.loads(resp.text) # or data = json.decoder(resp.text)
data