urllib.request 返回空数据，而 postman 中的相同请求返回正确数据答案

【问题标题】：The urllib.request returns empty data, while the same request in postman returns correct dataurllib.request 返回空数据，而 postman 中的相同请求返回正确数据
【发布时间】：2020-11-02 05:28:26
【问题描述】：

我的网址：

https://www.grants.gov/grantsws/rest/opportunities/search/

网址负载：

payload = { "startRecordNum":0,
            "sortBy":"openDate|desc",
            "oppStatuses":"forecasted|posted"
          }

网址标题：

headers = {'Accept':'application/json, text/javascript, */*; q=0.01',
                 'Content-Type':'application/json; charset=UTF-8' ,
                 'Origin':'https://www.grants.gov' , 
                 'Accept-Language':'en-US,en;q=0.9,fa-AF;q=0.8,fa;q=0.7,ru;q=0.6' }

我的 Python 代码：

import urllib.request
import urllib.parse

req = urllib.request.Request('https://www.grants.gov/grantsws/rest/opportunities/search/')
req.add_header('Accept','application/json, text/javascript, */*; q=0.01')
req.add_header('Content-Type','application/json; charset=UTF-8')
req.add_header('Origin','https://www.grants.gov')
req.add_header('Accept-Language','en-US,en;q=0.9,fa-AF;q=0.8,fa;q=0.7,ru;q=0.6')
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36')
payload = {"startRecordNum":0,
                "sortBy":"openDate|desc",
                "oppStatuses":"forecasted|posted"}
data = urllib.parse.urlencode(payload).encode()
# data = data.encode('ascii')


# r = urllib.request.urlopen(req)
with urllib.request.urlopen(req, data) as response:
    print(type(response))
    dataList = json.load(response)
    # searchParams = dataList['searchParams']
    # print(searchParams)
    print(dataList)

我的结果：

{'hitCount': 0, 'startRecord': 0, 'oppHits': [], 'oppStatusOptions': [], 'dateRangeOptions': [], 'suggestion': '', 'eligibilities': [], 'fundingCategories': [], 'fundingInstruments': [], 'agencies': [], 'accessKey': '', 'errorMsgs': []}

而我希望上述 dict 键的值不应该为空，因为我在 Postman 中使用 post 请求获得了正确的输出。

我应该怎么做才能得到正确的输出。如果您想探索请求和参数，这是链接...

enter link description here

【问题讨论】：

标签： python web-scraping python-requests urllib

【解决方案1】：

您需要使用json.dumps(payload) 将有效负载编码为json 格式：

import urllib.request
import json

req = urllib.request.Request('https://www.grants.gov/grantsws/rest/opportunities/search/')
req.add_header('Content-Type','application/json; charset=UTF-8')
payload = {
    "startRecordNum": 0,
    "sortBy":"openDate|desc",
    "oppStatuses":"forecasted|posted"
}
data = json.dumps(payload).encode()

with urllib.request.urlopen(req, data) as response:
    dataList = json.load(response)
    print(dataList)

【讨论】：