发送带参数的发布请求时脚本卡住答案

【问题标题】：Script gets stuck while sending post requests with parameters发送带参数的发布请求时脚本卡住
【发布时间】：2021-01-31 16:57:01
【问题描述】：

我正在尝试使用来自网页的适当参数来填充发出 post http 请求的 json 响应。当我运行脚本时，我看到脚本卡住并且没有带来任何结果。它也不会抛出任何错误。这是site link。在点击Get times & tickets 按钮之前，我从该站点的this form 的三个下拉列表中选择了三个选项。

我试过了：

import requests
from bs4 import BeautifulSoup

url = 'https://www.thetrainline.com/'
link = 'https://www.thetrainline.com/api/journey-search/'

payload = {"passengers":[{"dateOfBirth":"1991-01-31"}],"isEurope":False,"cards":[],"transitDefinitions":[{"direction":"outward","origin":"1f06fc66ccd7ea92ae4b0a550e4ddfd1","destination":"7c25e933fd14386745a7f49423969308","journeyDate":{"type":"departAfter","time":"2021-02-11T22:45:00"}}],"type":"single","maximumJourneys":4,"includeRealtime":True,"applyFareDiscounts":True}

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    s.headers['content-type'] = 'application/json'
    s.headers['accept'] = 'application/json'
    r = s.post(link,json=payload)
    print(r.status_code)
    print(r.json())

我怎样才能从该站点获取带有参数的发布请求的 json 响应？

【问题讨论】：

您可以通过将根日志级别设置为调试来查看完整的请求调试日志。我们很难在没有看到流量的情况下帮助回答这样的问题。另一种方法是使用 Fiddler 之类的工具来查看您的流量。

标签： python json python-3.x web-scraping python-requests

【解决方案1】：

您缺少必需的标题：x-version 和 referer。 referer 标头是指搜索表单，您可以构建它。在journey-search 之前，您必须发布availability 请求。

import requests
from requests.models import PreparedRequest

headers = {
    'authority': 'www.thetrainline.com',
    'pragma': 'no-cache',
    'cache-control': 'no-cache',
    'x-version': '2.0.18186',
    'dnt': '1',
    'accept-language': 'en-GB',
    'sec-ch-ua-mobile': '?0',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/88.0.4324.96 Safari/537.36',
    'content-type': 'application/json',
    'accept': 'application/json',
    'origin': 'https://www.thetrainline.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
}

with requests.Session() as s:
    origin = "6e2242b3f38bbbd8d8124e1d84d319e1"
    destination = "15bcf02bc44ea754837c8cf14569f608"
    localDateTime = "2021-02-03T19:30:00"
    dateOfBirth = "1991-02-03"
    passenger_type = "single"

    req = PreparedRequest()
    url = "http://www.neo4j.com"
    params = {
        "origin": origin,
        "destination": destination,
        "outwardDate": localDateTime,
        "outwardDateType": "departAfter",
        "journeySearchType": passenger_type,
        "passengers[]": dateOfBirth
    }
    req.prepare_url("https://www.thetrainline.com/book/results", params)

    headers.update({"referer": req.url})
    s.headers = headers

    payload_availability = {
        "origin": origin,
        "destination": destination,
        "outwardDefinition": {
            "localDateTime": localDateTime,
            "searchMethod": "DEPARTAFTER"
        },
        "passengerBirthDates": [{
            "id": "PASSENGER-0",
            "dateOfBirth": dateOfBirth
        }],
        "maximumNumberOfJourneys": 4,
        "discountCards": []
    }
    r = s.post('https://www.thetrainline.com/api/coaches/availability', json=payload_availability)
    r.raise_for_status()

    payload_search = {
        "passengers": [{"dateOfBirth": "1991-02-03"}],
        "isEurope": False,
        "cards": [],
        "transitDefinitions": [{
            "direction": "outward",
            "origin": origin,
            "destination": destination,
            "journeyDate": {
                "type": "departAfter",
                "time": localDateTime}
        }],
        "type": passenger_type,
        "maximumJourneys": 4,
        "includeRealtime": True,
        "applyFareDiscounts": True
    }
    r = s.post('https://www.thetrainline.com/api/journey-search/', json=payload_search)
    r.raise_for_status()

    print(r.json())

【讨论】：

【解决方案2】：

作为 Sers 的回复，标题丢失了。

在抓取网站时，您必须牢记防抓取机制。该网站会根据您的 IP 地址、请求标头、cookie 和各种其他因素来阻止您的请求。

【讨论】：