使用带有 mpi4py 的主机文件生成进程

Question

我正在尝试使用 MPI4py 和 OpenMPI 在多个主机上生成一组工作进程，但 spawn 命令似乎忽略了我的主机文件。我已经发布了my full test，但这里是关键部分：

基于a forum discussion，我的经理脚本使用hostfile 选项调用spawn：

mpi_info = MPI.Info.Create()
mpi_info.Set("hostfile", "worker_hosts")

comm = MPI.COMM_SELF.Spawn(sys.executable,
                           args=['testworker.py'],
                           maxprocs=args.worker_count,
                           info=mpi_info).Merge()

在worker_hosts 文件中，我列出了我的 Scyld Beowulf 集群中的节点：

myhead1 slots=2
mycompute1 slots=2
mycompute2 slots=2
mycompute3 slots=2
mycompute4 slots=3

经理和工人都打电话MPI.Get_processor_name()，但他们都报告“myhead1”。如果我使用与mpirun 相同的主机文件，它可以工作：

> mpirun -hostfile worker_hosts -np 3 python -c "from mpi4py import MPI; print MPI.Get_processor_name()"
myhead1
myhead1
mycompute1

如果我将主机文件的名称更改为不存在的名称，例如 bogus_file，则会收到错误消息：

--------------------------------------------------------------------------
Open RTE was unable to open the hostfile:
    bogus_file
Check to make sure the path and filename are correct.
--------------------------------------------------------------------------
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_support_fns.c at line 83
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file rmaps_rr.c at line 82
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/rmaps_base_map_job.c at line 88
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file base/plm_base_launch_support.c at line 105
[Bulbasaur:86523] [[3458,0],0] ORTE_ERROR_LOG: Not found in file plm_rsh_module.c at line 1173

所以 OpenMPI 注意到了 hostfile 选项，它似乎没有使用它。 hostfile 选项列在OpenMPI documentation 中。

Key                   Type      Description
---                   ----      -----------
host                  char *    Host on which the process should be spawned.
                                See the orte_host man page for an
                                explanation of how this will be used.
hostfile              char *    Hostfile containing the hosts on which
                                the processes are to be spawned. See
                                the orte_hostfile man page for an
                                explanation of how this will be used.

如何为生成请求指定主机文件？

Answer 1

我找到了OpenMPI documentation 的更新版本，它给了我神奇的选择：

Key                    Type     Description
---                    ----     -----------
host                   char *   Host on which the process should be
                                spawned.  See the orte_host man
                                page for an explanation of how this
                                will be used.
hostfile               char *   Hostfile containing the hosts on which
                                the processes are to be spawned. See
                                the orte_hostfile man page for
                                an explanation of how this will be
                                used.
add-host               char *   Add the specified host to the list of
                                hosts known to this job and use it for
                                the associated process. This will be
                                used similarly to the -host option.
add-hostfile           char *   Hostfile containing hosts to be added
                                to the list of hosts known to this job
                                and use it for the associated
                                process. This will be used similarly
                                to the -hostfile option.

如果我改为使用add-hostfile，它会完美运行：

mpi_info.Set("add-hostfile", "worker_hosts")

如果您无法使用旧版本的 OpenMPI，请尝试使用 mpirun 和相同的主机文件运行管理器脚本。当我仍在使用 hostfile 选项时，这也有效。

mpirun -hostfile worker_hosts -np1 python testmanager.py