无法使用 python 请求发送良好的查询字符串

Question

我正在尝试使用 python 和请求库对该 URL 发出正确的获取请求，并应用一些过滤器：

https://www.efast.dol.gov/5500search/

我只需要一些过滤器来在搜索页面中获取正确的数据，它们是：planyear、ein 和 pn。当我尝试执行请求时，我得到了错误的数据，因为我的 dict 在“q”之后得到了一个已删除的值

这是一个例子：

import requests

args = {'q.parser': 'lucene', 'q': {'ein': '814699012', 'planyear': '2020', 'pn': '001'}}
url = "https://www.efast.dol.gov/services/afs"
response = requests.get(url, params=args)

当我检查 response.url 时，我得到：

https://www.efast.dol.gov/services/afs?q.parser=lucene&q=ein&q=planyear&q=pn

每个键都没有价值

这是我最近的一次：

 args = {"q.parser":"lucene","q":{"ein":"814699012"}, "planyear":"2020","pn":"001"}

但是如果我做 response.url 我得到：

'https://www.efast.dol.gov/services/afs?q.parser=lucene&q=ein&planyear=2020&pn=001

ein值没了，不管我把planyear或者pn作为值放在q旁边，结果都是一样的。

我做错了什么？

正确的结果是2020年对应的数据，正确的ein数和pn数，我得到几个结果还是一个都没关系

正确的结果是这样的：

https://www.efast.dol.gov/services/afs?q.parser=lucene&size=200&sort=planname%20asc&q=(((planyear:2020))%20AND%20((ein:814699012))%20AND%20((pn:001)))&facet.planyear=%7Bsize:30%7D%26facet.plancode=%7Bsize:100%7D&facet.plancode=%7Bsize:100%7D&facet.assetseoy=%7Bbuckets:%5B%22%7B,100000%5D%22,%22%5B100001,500000%5D%22,%22%5B500001,1000000%5D%22,%22%5B1000001,10000000%5D%22,%22%5B10000001,%7D%22%5D%7D&facet.plantype=%7Bsize:20%7D&facet.businesscodecat=%7Bsize:30%7D&facet.businesscode=%7Bsize:30%7D&facet.state=%7Bsize:100%7D&facet.countrycode=%7Bbuckets:%5B%22CA%22,%22GB%22,%22BM%22,%22KY%22%5D%7D&facet.formyear=%7Bsize:30%7D

Answer 1

Python 的requests 包不支持类似字典的参数。它们在键值字典中的值必须是字符串或字符串列表：

Requests 允许您使用 params 关键字参数将这些参数作为字符串字典提供。

https://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls

您的网站正在使用非标准编码将字典编码为 url 有效字符。

如果你看看你的例子：

(((planyear:2020))%20AND%20((ein:814699012))%20AND%20((pn:001)))

我们可以推导出格式为：

(((KEY:VALUE)) AND ((KEY:VALUE)) AND <...>)

所以它是()，其中每个键：值对都被(())包围，空格被url引用到%20。

我们可以在我们的代码中自己复制这种编码：

>>> params = {"planyear": "2020", "ein": 814699012, "pn": "001"}
>>> encoded = '%20AND%20'.join(f"(({k}:{v})" for k, v in params.items())
>>> f"({encoded})"
'(((planyear:2020))%20AND%20((ein:814699012))%20AND%20((pn:001)))'

然后将其作为q 参数传递。

编辑：我已针对您的具体情况进行了编译：

import requests

q = {'ein': '814699012', 'planyear': '2020', 'pn': '001'}
# convert Q parameter to website's encoding: 
q = ' AND '.join(f"(({k}:{v}))" for k, v in q.items())
q = f"({q})"
# put all params together: normal key:value parameters + special Q parameter: 
params = {'q.parser': 'lucene', 'size': 200, 'sort': 'planname asc', 'q': q}
url = "https://www.efast.dol.gov/services/afs"
response = requests.get(url, params=params)

print(response.status)
print(response.url)
# 200
# 'https://www.efast.dol.gov/services/afs?q.parser=lucene&size=200&sort=planname+asc&q=%28%28%28ein%3A814699012%29%29+AND+%28%28planyear%3A2020%29%29+AND+%28%28pn%3A001%29%29%29'

Answer 2

您似乎对请求和响应感到困惑。 在您的情况下，您应该使用长 URL 作为您的请求，然后解析响应 JSON 数据。所以下面的代码应该适合你，你需要解析响应：

import requests

url = "https://www.efast.dol.gov/services/afs?q.parser=lucene&size=200&sort=planname%20asc&q=(((planyear:2020))%20AND%20((ein:814699012))%20AND%20((pn:001)))&facet.planyear=%7Bsize:30%7D%26facet.plancode=%7Bsize:100%7D&facet.plancode=%7Bsize:100%7D&facet.assetseoy=%7Bbuckets:%5B%22%7B,100000%5D%22,%22%5B100001,500000%5D%22,%22%5B500001,1000000%5D%22,%22%5B1000001,10000000%5D%22,%22%5B10000001,%7D%22%5D%7D&facet.plantype=%7Bsize:20%7D&facet.businesscodecat=%7Bsize:30%7D&facet.businesscode=%7Bsize:30%7D&facet.state=%7Bsize:100%7D&facet.countrycode=%7Bbuckets:%5B%22CA%22,%22GB%22,%22BM%22,%22KY%22%5D%7D&facet.formyear=%7Bsize:30%7D"

payload={}
headers = {}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)

检查一下：

import requests

parser='lucene'
query='(((planyear:2020))%20AND%20((ein:814699012))%20AND%20((pn:001)))'

url = "https://www.efast.dol.gov/services/afs?q.parser=" + parser + "&size=200&sort=planname%20asc&q=" + query + "&facet.planyear=%7Bsize:30%7D%26facet.plancode=%7Bsize:100%7D&facet.plancode=%7Bsize:100%7D&facet.assetseoy=%7Bbuckets:%5B%22%7B,100000%5D%22,%22%5B100001,500000%5D%22,%22%5B500001,1000000%5D%22,%22%5B1000001,10000000%5D%22,%22%5B10000001,%7D%22%5D%7D&facet.plantype=%7Bsize:20%7D&facet.businesscodecat=%7Bsize:30%7D&facet.businesscode=%7Bsize:30%7D&facet.state=%7Bsize:100%7D&facet.countrycode=%7Bbuckets:%5B%22CA%22,%22GB%22,%22BM%22,%22KY%22%5D%7D&facet.formyear=%7Bsize:30%7D"

payload={}
headers = {}

response = requests.request("GET", url, headers=headers, data=payload)

print(response.text)