带有 SSH 的 Ipython 并行 LAN - 找不到 ipengine答案

【问题标题】：Ipython parallel LAN with SSH - ipengine cannot be found带有 SSH 的 Ipython 并行 LAN - 找不到 ipengine
【发布时间】：2015-08-11 22:13:07
【问题描述】：

我希望有人能对这个问题有所了解，谷歌搜索结果真的一无所获，这本身就是一个明显的迹象，表明我在做一些完全错误的事情。无论如何，这是我的设置。相当标准的尝试在 LAN 中与另外两台 ubuntu 14.04 服务器机器一起运行 ipcluster。

将ipcluster_config.py修改为

     c.IPClusterEngines.engine_launcher_class = 'SSH'
     c.SSHEngineSetLauncher.engines = { 'hp4@192.168.1.214'  : 2,
                               'hp3@192.168.1.213' : 2 }

修改ipcontroller_config.py

     c.HubFactory.ip =  '*'

从客户端和控制器机器启动集群

     ipcluster start --profile=ssh --debug

现在它启动了集群，引擎尝试连接，然后立即出错说

     2015-08-11 17:44:54.757 [IPClusterStart] bash: ipengine: command not

找到

我可以手动登录到那些机器并执行ipengine 命令它连接没有问题，所以我错过了什么

感谢您的输入，下面有更详细的日志

2015-08-11 17:44:50.494 [IPClusterStart] Config changed:
2015-08-11 17:44:50.494 [IPClusterStart] {'Application': {'log_level': 10}, 'BaseIPythonApplication': {'profile': u'ssh'}}
2015-08-11 17:44:50.494 [IPClusterStart] IPYTHONDIR set to: /home/d752/.ipython
2015-08-11 17:44:50.500 [IPClusterStart] Using existing profile dir: u'/home/d752/.ipython/profile_ssh'
2015-08-11 17:44:50.501 [IPClusterStart] Searching path [u'/home/d752/JP/ipythonwork', u'/home/d752/.ipython/profile_ssh', '/usr/local/etc/ipython', '/etc/ipython'] for config files
2015-08-11 17:44:50.501 [IPClusterStart] Attempting to load config file: ipython_config.py
2015-08-11 17:44:50.502 [IPClusterStart] Loaded config file: /home/d752/.ipython/profile_ssh/ipython_config.py
2015-08-11 17:44:50.502 [IPClusterStart] Attempting to load config file: ipcluster_config.py
2015-08-11 17:44:50.503 [IPClusterStart] Loaded config file: /home/d752/.ipython/profile_ssh/ipcontroller_config.py
2015-08-11 17:44:50.504 [IPClusterStart] Config changed:
2015-08-11 17:44:50.504 [IPClusterStart] {'Application': {'log_level': 10}, 'BaseIPythonApplication': {'profile': u'ssh'}, 'HubFactory': {'ip': '*'}}
2015-08-11 17:44:50.505 [IPClusterStart] Attempting to load config file: ipcluster_config.py
2015-08-11 17:44:50.505 [IPClusterStart] Loaded config file: /home/d752/.ipython/profile_ssh/ipengine_config.py
2015-08-11 17:44:50.506 [IPClusterStart] Attempting to load config file: ipcluster_config.py
2015-08-11 17:44:50.507 [IPClusterStart] Loaded config file: /home/d752/.ipython/profile_ssh/ipcluster_config.py
2015-08-11 17:44:50.508 [IPClusterStart] Config changed:
2015-08-11 17:44:50.508 [IPClusterStart] {'Application': {'log_level': 10}, 'BaseIPythonApplication': {'profile': u'ssh'}, 'HubFactory': {'ip': '*'}, 'IPClusterEngines': {'engine_launcher_class': 'SSH'}, 'SSHEngineSetLauncher': {'engines': {'hp4@192.168.1.214': 2, 'hp3@192.168.1.213': 2}}}
2015-08-11 17:44:50.511 [IPClusterStart] Starting ipcluster with [daemon=False]
2015-08-11 17:44:50.512 [IPClusterStart] Creating pid file: /home/d752/.ipython/profile_ssh/pid/ipcluster.pid
2015-08-11 17:44:50.512 [IPClusterStart] Starting Controller with LocalControllerLauncher
2015-08-11 17:44:50.512 [IPClusterStart] Starting LocalControllerLauncher: ['/home/d752/anaconda/bin/python', '-m', 'IPython.parallel.controller', '--profile-dir', u'/home/d752/.ipython/profile_ssh', '--cluster-id', u'', '--log-to-file', '--log-level=20']
2015-08-11 17:44:50.515 [IPClusterStart] Process '/home/d752/anaconda/bin/python' started: 14930
2015-08-11 17:44:50.889 [IPClusterStart] 2015-08-11 17:44:50.888 [IPControllerApp] Using existing profile dir: u'/home/d752/.ipython/profile_ssh'
2015-08-11 17:44:51.174 [IPClusterStart] 2015-08-11 17:44:51.174 [scheduler] Scheduler started [leastload]
2015-08-11 17:44:51.516 [IPClusterStart] Starting 4 Engines with SSH
2015-08-11 17:44:51.518 [IPClusterStart] ensuring remote hp3@192.168.1.213:.ipython/profile_ssh/security/ exists
Connection to 192.168.1.213 closed.
2015-08-11 17:44:51.901 [IPClusterStart] sending /home/d752/.ipython/profile_ssh/security/ipcontroller-client.json to hp3@192.168.1.213:.ipython/profile_ssh/security/ipcontroller-client.json
2015-08-11 17:44:52.273 [IPClusterStart] ensuring remote hp3@192.168.1.213:.ipython/profile_ssh/security/ exists
Connection to 192.168.1.213 closed.
2015-08-11 17:44:52.654 [IPClusterStart] sending /home/d752/.ipython/profile_ssh/security/ipcontroller-engine.json to hp3@192.168.1.213:.ipython/profile_ssh/security/ipcontroller-engine.json
2015-08-11 17:44:53.030 [IPClusterStart] Starting SSHEngineLauncher: ['ssh', '-tt', u'hp3@192.168.1.213', 'ipengine', '--profile-dir', u'.ipython/profile_ssh', '--log-to-file', '--log-level=20']
2015-08-11 17:44:53.034 [IPClusterStart] Process 'ssh' started: 14955
2015-08-11 17:44:53.136 [IPClusterStart] Starting SSHEngineLauncher: ['ssh', '-tt', u'hp3@192.168.1.213', 'ipengine', '--profile-dir', u'.ipython/profile_ssh', '--log-to-file', '--log-level=20']
2015-08-11 17:44:53.139 [IPClusterStart] Process 'ssh' started: 14956
2015-08-11 17:44:53.141 [IPClusterStart] ensuring remote hp4@192.168.1.214:.ipython/profile_ssh/security/ exists
Connection to 192.168.1.214 closed.
2015-08-11 17:44:53.516 [IPClusterStart] sending /home/d752/.ipython/profile_ssh/security/ipcontroller-client.json to hp4@192.168.1.214:.ipython/profile_ssh/security/ipcontroller-client.json
2015-08-11 17:44:53.892 [IPClusterStart] ensuring remote hp4@192.168.1.214:.ipython/profile_ssh/security/ exists
Connection to 192.168.1.214 closed.
2015-08-11 17:44:54.271 [IPClusterStart] sending /home/d752/.ipython/profile_ssh/security/ipcontroller-engine.json to hp4@192.168.1.214:.ipython/profile_ssh/security/ipcontroller-engine.json
2015-08-11 17:44:54.648 [IPClusterStart] Starting SSHEngineLauncher: ['ssh', '-tt', u'hp4@192.168.1.214', 'ipengine', '--profile-dir', u'.ipython/profile_ssh', '--log-to-file', '--log-level=20']
2015-08-11 17:44:54.651 [IPClusterStart] Process 'ssh' started: 14963
2015-08-11 17:44:54.754 [IPClusterStart] Starting SSHEngineLauncher: ['ssh', '-tt', u'hp4@192.168.1.214', 'ipengine', '--profile-dir', u'.ipython/profile_ssh', '--log-to-file', '--log-level=20']
2015-08-11 17:44:54.757 [IPClusterStart] Process 'ssh' started: 14964
2015-08-11 17:44:54.757 [IPClusterStart] Process 'engine set' started: [None, None, None, None]
2015-08-11 17:44:54.757 [IPClusterStart] bash: ipengine: command not found
2015-08-11 17:44:54.758 [IPClusterStart] Connection to 192.168.1.213 closed.
2015-08-11 17:44:54.758 [IPClusterStart] bash: ipengine: command not found
2015-08-11 17:44:54.758 [IPClusterStart] Connection to 192.168.1.213 closed.
2015-08-11 17:44:54.758 [IPClusterStart] Process 'ssh' stopped: {'pid': 14955, 'exit_code': 127}
2015-08-11 17:44:54.758 [IPClusterStart] Process 'ssh' stopped: {'pid': 14956, 'exit_code': 127}
2015-08-11 17:44:55.026 [IPClusterStart] bash: ipengine: command not found
2015-08-11 17:44:55.026 [IPClusterStart] Connection to 192.168.1.214 closed.
2015-08-11 17:44:55.026 [IPClusterStart] Process 'ssh' stopped: {'pid': 14963, 'exit_code': 127}
2015-08-11 17:44:55.130 [IPClusterStart] bash: ipengine: command not found
2015-08-11 17:44:55.130 [IPClusterStart] Connection to 192.168.1.214 closed.
2015-08-11 17:44:55.130 [IPClusterStart] Process 'ssh' stopped: {'pid': 14964, 'exit_code': 127}
2015-08-11 17:44:55.130 [IPClusterStart] Process 'engine set' stopped: {'192.168.1.213/0': {'pid': 14955, 'exit_code': 127}, '192.168.1.213/1': {'pid': 14956, 'exit_code': 127}, '192.168.1.214/1': {'pid': 14964, 'exit_code': 127}, '192.168.1.214/0': {'pid': 14963, 'exit_code': 127}}
2015-08-11 17:44:55.130 [IPClusterStart] ERROR | 
            Engines shutdown early, they probably failed to connect.

            Check the engine log files for output.

            If your controller and engines are not on the same machine, you probably
            have to instruct the controller to listen on an interface other than localhost.

            You can set this by adding "--ip='*'" to your ControllerLauncher.controller_args.

            Be sure to read our security docs before instructing your controller to listen on
            a public interface.

2015-08-11 17:44:55.131 [IPClusterStart] ERROR | IPython cluster: stopping
2015-08-11 17:44:55.163 [IPClusterStart] Process '/home/d752/anaconda/bin/python' stopped: {'pid': 14930, 'exit_code': 0}
2015-08-11 17:44:58.132 [IPClusterStart] Removing pid file: /home/d752/.ipython/profile_ssh/pid/ipcluster.pid

【问题讨论】：

标签： ssh parallel-processing ipython

【解决方案1】：

我遇到了同样的问题，这就是我解决它的方法。

就我而言，我的 ipcluster_config.py 具有以下设置：

c.SSHEngineSetLauncher.engines = {
 'host1' : 4,
 'host2' : 8,
 'host3' : 4,
}

在每个服务器中都没有找到 ipengine 启动器。我在所有机器（控制器和客户端）中为 ipengine 创建了一个别名：

sudo ln -s PATH_TO_IPENGINE_BINARY /opt/ipengine

在我的例子中，它指向 Anaconda ipengine。

然后我在 ipcluster_config.py 中更改了以下变量：

c.SSHEngineSetLauncher.engine_cmd = ['/opt/ipengine']

如果你在不同的客户端有不同的用户名，在控制器机器中创建一个 .ssh/config 文件来设置一切是非常方便的。例如：

Host host1
  HostName xx.xxx.x.xxx
  User me

Host host2
  HostName xx.xxx.x.xxx
  User myself

Host host3
  HostName xx.xxx.x.xxx
  User irene

另外，请确保您已在每台机器上配置无密码 SSH 登录。现在应该可以了，我希望。

【讨论】：