多处理线程池在最后停止答案

【问题标题】：multiprocessing ThreadPool stops at the end多处理线程池在最后停止
【发布时间】：2020-09-13 18:40:42
【问题描述】：

我编写了一个脚本来“解析”文件中的所有域。启动后，一切正常。但是当最后剩下几个域时，它就会卡住。有时解析最后几个域需要很长时间。我无法弄清楚问题是什么。谁遇到过这样的情况？告诉我如何治愈它。

发布后，一切都很快（应该如此）顺利进行，直到最后。最后，当剩下几个域时它会停止。 1000 域或 10 000 域没有区别。

完整代码：

import re
import sys
import json
import requests
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

pool = 100

with open("Rules.json") as file:
    REGEX = json.loads(file.read())

ua = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0'}

def Domain_checker(domain):
    try:
        r = requests.get("http://" + domain, verify=False, headers=ua)
        r.encoding = "utf-8"

        for company in REGEX.keys():

            for type in REGEX[company]:
                check_entry = 0

                for ph_regex in REGEX[company][type]:
                    if bool(re.search(ph_regex, r.text)) is True:
                        check_entry += 1

                        if check_entry == len(REGEX[company][type]):
                            title = BeautifulSoup(r.text, "lxml")
                            Found_domain = "\nCompany: {0}\nRule: {1}\nURL: {2}\nTitle: {3}\n".format(company, type, r.url, title.title.text)
                            print(Found_domain)
                            with open("/tmp/__FOUND_DOMAINS__.txt", "a", encoding='utf-8', errors = 'ignore') as file:
                                file.write(Found_domain)

    except requests.exceptions.ConnectionError:
        pass
    except requests.exceptions.TooManyRedirects:
        pass
    except requests.exceptions.InvalidSchema:
        pass
    except requests.exceptions.InvalidURL:
        pass
    except UnicodeError:
        pass
    except requests.exceptions.ChunkedEncodingError:
        pass
    except requests.exceptions.ContentDecodingError:
        pass
    except AttributeError:
        pass
    except ValueError:
        pass

    return domain


if __name__ == '__main__':

    with open(sys.argv[1], "r", encoding='utf-8', errors = 'ignore') as file:
        Domains = file.read().split()

    pool = 100
    print("Pool = ", pool)

    results = ThreadPool(pool).imap_unordered(Domain_checker, Domains)
    string_num = 0

    for result in results:
        print("{0} => {1}".format(string_num, result))
        string_num += 1

    with open("/tmp/__FOUND_DOMAINS__.txt", encoding='utf-8', errors = 'ignore') as found_domains:
        found_domains = found_domains.read()

    print("{0}\n{1}".format("#" * 40, found_domains))

【问题讨论】：

除了块之外的其中一个可能会抑制正在引发的异常。至少打印异常。
异常中没有因为ThreadPool的错误。仅与域不可用、编码等相关的异常
Alligat0r：由于您正在抑制所有这些异常，您无法知道是什么导致了它们。顺便说一句，您可以在一个except 子句中以相同的方式处理多个异常，方法是创建它们的元组：即except (requests.exceptions.ConnectionError, requests.exceptions.TooManyRedirects, etc...):。

标签： python multiprocessing threadpool

【解决方案1】：

requests.get("http://" + domain, headers=ua, verify=False, timeout=10)

安装后问题解决timeout

感谢昵称“eri”的用户 (https://ru.stackoverflow.com/users/16574/eri) :)

【讨论】：