如何处理 urllib.error.URLError: <urlopen 错误 [WinError 10054] 现有连接被远程主机强行关闭>？答案

【问题标题】：How to handle urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>?如何处理 urllib.error.URLError: <urlopen 错误 [WinError 10054] 现有连接被远程主机强行关闭>？
【发布时间】：2021-08-02 12:46:25
【问题描述】：

我有以下一小段 python 脚本：

from urllib.request import urlopen

def download_file(url):
    fp = open(url.split("=")[-1] + ".pdf", 'wb')
    req = urlopen(url)
    CHUNK = 20480
    chunk = req.read(CHUNK)
    fp.write(chunk)
    fp.close()

for i in range(1, 10000, 1):
    download_file("__some__url__" + str(i))

print("Done.")

我保持这个脚本运行，但由于某种原因，一段时间后（比如说在下载 100 个文件之后）它给出了一个错误：urllib.error.URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>

如何修改我的代码以处理该错误，即它不应该停止脚本和处理，即等待连接恢复，然后从它离开的地方继续下载？

PS：我知道它只从 URL 下载 20KB。

【问题讨论】：

标签： python urllib

【解决方案1】：

有关此错误的可能原因，您可以查看python: [Errno 10054] An existing connection was forcibly closed by the remote host。阅读最高投票答案后，我的结论是它可能会发生，您的代码应该为此做好准备。

我会在这里循环使用try: ... except ... 块，并在重试失败的连接之前增加延迟：

def download_file(url):
    # prefere with when exceptions are expected
    with open(url.split("=")[-1] + ".pdf", 'wb') as fp:
        delay = 5
        max_retries = 3
        for _ in range(max_retries):
            try:
                req = urlopen(url)
                CHUNK = 20480
                chunk = req.read(CHUNK)
                fp.write(chunk)
                break                # do not loop after a successfull download...
            except urllib.error.URLError:
                time.sleep(delay)
                delay *= 2
        else:                # signal an abort if download was not possible
            print(f"Failed for {url} after {max_retries} attempt")
            # or if you want to abort the script
            # raise Exception(f"Failed for {url} after {max_retries} attempt")

【讨论】：