使用 requests.get(URL) 获取 (requests.exceptions.ConnectionError)答案

【问题标题】：Getting (requests.exceptions.ConnectionError) using requests.get(URL)使用 requests.get(URL) 获取 (requests.exceptions.ConnectionError)
【发布时间】：2021-09-18 17:49:10
【问题描述】：

运行以下代码时出现requests.exceptions.ConnectionError 错误：

from requests import *
from bs4 import *

URL = "https://www.ldoceonline.com/dictionary/"
response = get(URL)

但是当我用另一个 URL 测试它时，它可以工作。我真的很想刮掉这个网站。如何解决此错误？

完整的错误说明：

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "e:/amirhossein/Project/Programming/L/Longmandict.py", line 5, in <module>
    response = get(URL,verify=False)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

【问题讨论】：

标签： python web-scraping python-requests

【解决方案1】：

显然服务器需要设置正确的User-Agent HTTP 标头：

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"
}

url = "https://www.ldoceonline.com/dictionary/"
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")

print(soup.title.text)

打印：

Longman English Dictionaries | Meanings, thesaurus, collocations and grammar

【讨论】：