【问题标题】:Why is my function not printing the string in multiprocessing?为什么我的函数不在多处理中打印字符串?
【发布时间】:2021-12-13 14:39:41
【问题描述】:

我尝试深入了解多处理。到目前为止一切顺利,我理解了这个概念。但是现在我想知道,为什么在使用多处理时我的打印语句没有显示出来。

有谁知道我的错误在哪里或者为什么 print 参数没有出现在多处理中?

这是我的代码和输出,没有多处理:

# -------------------------LINEAR PROCESSING--------------------------- # 
import time

start = time.perf_counter()

def sleep(seconds):
    print("Sleeping {} second(s) ...".format(seconds))
    time.sleep(seconds)
    print("Done Sleeping...")

# run sleep function 10 times
for _ in range(10): # _ throw away variable - hence not using integers of range
    sleep(1.5)

finish = time.perf_counter()

print("Finished in {} second(s) without multi-processing".format(round(finish-start,2)))

# Output
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Sleeping 1.5 second(s) ...
Done Sleeping...
Finished in 15.03 second(s) without multi-processing

这是我的代码和多处理输出:

# -------------------------MULTI-PROCESSING (OLD WAY)--------------------------- # 
import multiprocessing
import time

start = time.perf_counter()

def sleep(seconds):
    print("Sleeping {} second(s) ...".format(seconds))
    time.sleep(seconds)
    print("Done Sleeping...")

# create 10 processes for each sleep function and store it in list
processes = []
for _ in range(10): # _ throw away variable - hence not using integers of range
    p = multiprocessing.Process(target=sleep, args=[1.5])
    p.start()
    processes.append(p)

# loop over started processes and wait until all processes are finished (join)
for process in processes:
    process.join()
    

finish = time.perf_counter()

print("Finished in {} second(s) with multi-processing".format(round(finish-start,2)))

# Output
Finished in 0.14 second(s) with multi-processing

这是我的 jupyter notebook 统计数据:

jupyter core     : 4.7.1
jupyter-notebook : 6.3.0
qtconsole        : 5.0.3
ipython          : 7.22.0
ipykernel        : 5.3.4
jupyter client   : 6.1.12
jupyter lab      : 3.0.14
nbconvert        : 6.0.7
ipywidgets       : 7.6.3
nbformat         : 5.1.3
traitlets        : 5.0.5

【问题讨论】:

  • 不是 100% 确定,但我认为每个进程都有自己的 stdout,而不是与主脚本相同的输出。要从不同的进程读取标准输出,您需要重定向或读取这些流。类似的问题stackoverflow.com/questions/30793624/…
  • 或者不使用打印,而是将字符串输出到队列,然后主脚本可以读取该队列并输出到控制台。
  • 谢谢!但是,问题不在于使用 __ name __ == "__ main __": 行,因为该脚本从另一个脚本调用了系统调用,其中也有时间模块并且没有被保护。
  • @scotty3785 标准输出是从主进程复制的,但 jupyter 会重新定向(和后处理)标准输出并且不会告诉孩子们。这就是为什么 child print 在系统终端上可以正常工作而没有涉及重定向 tomfoolery 的原因。这在 pycharm 和其他一些人中也很常见。

标签: python-3.x multiprocessing python-multiprocessing


【解决方案1】:

我找到了问题的答案(某种程度上):

首先,我在 Visual Studio 中重新运行我的代码。在那里我发现,当我启动进程时,它并没有派生子进程。

Exception has occurred: RuntimeError       (note: full exception trace is shown but execution is paused at: <module>)

        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
  File "/Users/philipp/Desktop/Processing.py", line 18, in <module>
    p.start()
  File "<string>", line 1, in <module> (Current frame)

深入挖掘后,我发现只有在使用 Windows 操作系统(我使用的是 IOS)时才需要使用 freeze_support()。

但是,使用 __ name __ == "__ main __": 参数并将多处理放在其中就可以了。这是重构后的代码,它适用于具有给定输出的 Visual Studio 代码:

# -------------------------MULTI-PROCESSING (OLD WAY)--------------------------- # 
import multiprocessing
import time

start = time.perf_counter()

def sleep(seconds):
    print("Sleeping {} second(s) ...".format(seconds))
    time.sleep(seconds)
    print("Done Sleeping...")

if __name__ == "__main__":

    # create 10 processes for each sleep function and store it in list
    processes = []
    for _ in range(10): # _ throw away variable - hence not using integers of range
        p = multiprocessing.Process(target=sleep, args=[1])
        p.start()
        processes.append(p)

    # loop over started processes and wait until all processes are finished (join)
    for process in processes:
        process.join()
            
    finish = time.perf_counter()

    print("Finished in {} second(s) with multi-processing".format(round(finish-start,2)))

# Output
Sleeping 1 second(s) ...
Sleeping 1 second(s) ...
Sleeping 1 second(s) ...Sleeping 1 second(s) ...

Sleeping 1 second(s) ...
Sleeping 1 second(s) ...Sleeping 1 second(s) ...
Sleeping 1 second(s) ...
Sleeping 1 second(s) ...

Sleeping 1 second(s) ...
Done Sleeping...
Done Sleeping...
Done Sleeping...
Done Sleeping...
Done Sleeping...
Done Sleeping...Done Sleeping...

Done Sleeping...
Done Sleeping...
Done Sleeping...
Finished in 2.51 second(s) with multi-processing

一些备注:

  • 这仅适用于 Visual Studio 代码(在 jupyter notebook 中它仍然无法运行 - 似乎 jupyter notebook 根本无法识别多处理模块,因此快速完成时间

  • 非常重要:起初没有 __ name __ == "__ main __" 的无保护脚本调用了另一个无保护脚本,因为它也有模块时间。因此,后一个脚本也启动了系统调用。这不知何故导致了一个关键部分。

【讨论】:

  • 很高兴您能够回答自己的问题!这里有一些额外的想法:对“spawn vs fork”之间的区别做一些额外的阅读,了解为什么在某些情况下需要if __name__ == "__main__":。在使用交互式提示(如 jupyter)时为什么 spawn 并不总是正常工作也很重要。最后,freeze_support 仅在 windows 当您打算使用 py2exe 之类的东西 将脚本转换为 windows PE 格式时才需要
  • 谢谢!我将更深入地研究这个主题。
猜你喜欢
  • 2021-11-23
  • 1970-01-01
  • 2016-07-11
  • 2018-08-30
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2015-08-26
相关资源
最近更新 更多