使用请求导入数据时出错 - python

Question

我正在尝试从以下网页访问数据：https://qships.tmr.qld.gov.au/webx/# 使用“船舶移动”选项卡。 我尝试使用 url 服务请求，但不断收到错误。

目前，我已经尝试过：

import requests
import pandas as pd
import json

url = 'https://qships.tmr.qld.gov.au/webx/services/wxdata.svc/GetDataX'

payload = {
"token": None,
"reportCode": "MSQ-WEB-0001",
"dataSource": None,
"filterName": "Last 7 days",
"parameters": [{
        "__type": "ParameterValueDTO:#WebX.Core.DTO",
        "sName": "DOMAIN_ID",
        "iValueType": 0,
        "aoValues": [{"Value": -1}],
              }],
"metaVersion": 0,
}

jsonData = requests.post(url, data = payload).json()

返回以下错误：

{'ExceptionDetail': {'HelpLink': None, 'InnerException': None, 'Message': "The incoming message has an unexpected message format 'Raw'. The expected message formats for the operation are 'Xml', 'Json'. This can be because a WebContentTypeMapper has not been configured on the binding. See the documentation of WebContentTypeMapper for more details.", 'StackTrace': '   at System.ServiceModel.Dispatcher.DemultiplexingDispatchMessageFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.UriTemplateDispatchFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)', 'Type': 'System.InvalidOperationException'}, 'ExceptionType': 'System.InvalidOperationException', 'Message': "The incoming message has an unexpected message format 'Raw'. The expected message formats for the operation are 'Xml', 'Json'. This can be because a WebContentTypeMapper has not been configured on the binding. See the documentation of WebContentTypeMapper for more details.", 'StackTrace': '   at System.ServiceModel.Dispatcher.DemultiplexingDispatchMessageFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.UriTemplateDispatchFormatter.DeserializeRequest(Message message, Object[] parameters)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)\r\n   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)'}

当我更改 json 字典时，我收到以下错误：

jsonData = requests.post(url, json={'key':'value'}).json()

{'ExceptionDetail': None, 'ExceptionType': None, 'Message': 'WebX.Security.EAuthError: Request does not belong to an authenticated session.', 'StackTrace': None}

数据可以通过请求访问还是我必须抓取它？

Answer 1

按照链接的 Web UI 发出的请求，此脚本似乎能够访问 一些 数据。

数据查询的 JSON 负载与原始帖子不同，因此可能需要在此处进行一些调整。

无论如何，关键在于您需要使用requests.Session()，并首先向 Web UI 发出请求以获取会话 cookie。

from pprint import pprint

import requests

get_session_url = "https://qships.tmr.qld.gov.au/webx/"
get_data_url = "https://qships.tmr.qld.gov.au/webx/services/wxdata.svc/GetDataX"
get_data_query = {
    "token": None,
    "reportCode": "MSQ-WEB-0001",
    "dataSource": None,
    "filterName": "Next 7 days",
    "parameters": [
        {
            "__type": "ParameterValueDTO:#WebX.Core.DTO",
            "sName": "DOMAIN_ID",
            "iValueType": 0,
            "aoValues": [{"Value": -1}],
        }
    ],
    "metaVersion": 0,
}

sess = requests.session()

sess.get(get_session_url).raise_for_status()

json_data = sess.post(
    get_data_url,
    json=get_data_query,
).json()

pprint(json_data)

打印出来（例如）

{'d': {'BuildVersion': '7.0.0.12590',
       'ReportCode': 'MSQ-WEB-0001',
       'Tables': [{'AsOfDate': '16:33 on Jan 17',
                   'BuildVersion': '7.0.0.12590',
                   'Data': [[132058,
                             334359,
                             'EXT',
                             'STOLT MOMIJI',
                             'TANKER',
                             121.52,
...