多处理池意外杀死线程

Question

我在下面的 python 3.6.3 中编写并执行了一个简单的测试程序。它在一台 4 核的机器上执行。

import multiprocessing
import time

def f(num):
  print(multiprocessing.current_process(), num)
  time.sleep(1)
  if (num % 2):
    raise Exception


pool = multiprocessing.Pool(5)

try:
  pool.map(f, range(1,20))
except Exception as e:
  print("EXCEPTION")

pool.close()
pool.join()

输出pool = multiprocessing.Pool(5):

<ForkProcess(ForkPoolWorker-1, started daemon)> 1
<ForkProcess(ForkPoolWorker-2, started daemon)> 2
<ForkProcess(ForkPoolWorker-3, started daemon)> 3
<ForkProcess(ForkPoolWorker-4, started daemon)> 4
<ForkProcess(ForkPoolWorker-5, started daemon)> 5
<ForkProcess(ForkPoolWorker-2, started daemon)> 6
<ForkProcess(ForkPoolWorker-1, started daemon)> 7
<ForkProcess(ForkPoolWorker-4, started daemon)> 8
<ForkProcess(ForkPoolWorker-3, started daemon)> 9
<ForkProcess(ForkPoolWorker-5, started daemon)> 10
<ForkProcess(ForkPoolWorker-2, started daemon)> 11
<ForkProcess(ForkPoolWorker-1, started daemon)> 12
<ForkProcess(ForkPoolWorker-4, started daemon)> 13
<ForkProcess(ForkPoolWorker-3, started daemon)> 14
<ForkProcess(ForkPoolWorker-5, started daemon)> 15
<ForkProcess(ForkPoolWorker-2, started daemon)> 16
<ForkProcess(ForkPoolWorker-1, started daemon)> 17
<ForkProcess(ForkPoolWorker-3, started daemon)> 18
<ForkProcess(ForkPoolWorker-4, started daemon)> 19
EXCEPTION

但是，如果我将池的进程计数更改为等于或小于我机器上的内核数，则对 f() 的每次调用都不会打印，其中 num 是。

使用pool = multiprocessing.Pool(4) 输出：

<ForkProcess(ForkPoolWorker-1, started daemon)> 1
<ForkProcess(ForkPoolWorker-2, started daemon)> 3
<ForkProcess(ForkPoolWorker-3, started daemon)> 5
<ForkProcess(ForkPoolWorker-2, started daemon)> 7
<ForkProcess(ForkPoolWorker-1, started daemon)> 9
<ForkProcess(ForkPoolWorker-3, started daemon)> 11
<ForkProcess(ForkPoolWorker-3, started daemon)> 13
<ForkProcess(ForkPoolWorker-1, started daemon)> 15
<ForkProcess(ForkPoolWorker-2, started daemon)> 17
<ForkProcess(ForkPoolWorker-1, started daemon)> 19
EXCEPTION

我不明白为什么这些进程会被杀死，尤其是在函数中的 print 语句之后甚至没有抛出异常时。我真的不明白为什么只有当池中的进程数等于或小于机器上的内核数时才会发生这种情况。

Answer 1

参考multiprocessing.Pool.map的规范 您可以看到一个可选参数chunksize，如果您将其指定为1，即pool.map(f, range(1,20), 1)，那么您将产生预期的结果。

如果您增加块大小（例如 = 6），您可能会看到：

<SpawnProcess(SpawnPoolWorker-1, started daemon)> 1
<SpawnProcess(SpawnPoolWorker-4, started daemon)> 7
<SpawnProcess(SpawnPoolWorker-3, started daemon)> 13
<SpawnProcess(SpawnPoolWorker-2, started daemon)> 19

这表明chunksize的任务数被分配给Pool中的单个线程，当您在每个线程期间引发异常时，剩余卡盘中的任务当然不会执行。

从这里你可以知道chunksize的默认值在你的情况下是2，这个变量存在的原因，很容易看出，是为了减少需要初始化的新线程的数量（当您有适当的块大小时，这可能会节省资源和处理时间）。