两个MPI计算节点无法完成一次TCP连接，防火墙导致答案

【问题标题】：Two MPI computing nodes cannot complete a TCP connection cause by firewall两个MPI计算节点无法完成一次TCP连接，防火墙导致
【发布时间】：2021-06-25 11:40:10
【问题描述】：

我正在尝试在两个计算节点 node1 和 node2 上运行一个简单的 MPI 示例，它们是我刚刚在 Oracle Cloud 上创建的虚拟机。（这是我第一次使用Oracle Cloud...）系统是Ubuntu 20.04。我所做的包括：

node1 和 node2 在同一路径下具有正确的 MPI 环境 (OpenMPI-4.1.0)。 $PATH 和 $LD_LIBRARY_PATH 也已设置。我可以在单个节点上成功运行 MPI 示例。
已设置node1 和node2 之间的无密码登录。我可以使用ssh node1 和ssh node2 将一个节点连接到另一个节点。
在同一路径（$HOSTFILE_PATH/hosts2）下的两个节点上有一个主机文件（hosts2）包含

node1  slots=1
node2  slots=1

可执行文件 (test) 位于同一路径 ($EXE_PATH/test) 下。

然后我跑$(which mpirun) -n 2 -hostfile $HOSTFILE_PATH/hosts2 $EXEC_PATH/test，并没有得到任何回报。所以我只能用 ctrl+c 来终止执行。几分钟后，我得到了输出：

 ------------------------------------------------------------
A process or daemon was unable to complete a TCP connection
to another process:
  Local host:    instance-1-632783
  Remote host:   instance-1
This is usually caused by a firewall on the remote host. Please
check that any firewall (e.g., iptables) has been disabled and
try again.
------------------------------------------------------------

问题与防火墙有关吗？我尝试了sudo ufw status 并得到了Status: inactive。我也试过sudo iptables -L，得到了：

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp spt:ntp
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
InstanceServices  all  --  anywhere             link-local/16       

Chain InstanceServices (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             169.254.0.2          owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.2.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.4.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.5.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.2          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.3          owner UID match root tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.4          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:bootps /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:tftp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:ntp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
REJECT     tcp  --  anywhere             link-local/16        tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset
REJECT     udp  --  anywhere             link-local/16        udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable

然后我试了sudo iptables -F，之后sudo iptables -L显示：

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain InstanceServices (0 references)
target     prot opt source               destination

但似乎sudo iptables -F 暂时删除了这些策略。当我重新启动系统时，sudo iptables -L 显示以前的输出。那么如何解决防火墙问题呢？我应该永久删除这些政策吗？怎么做？

【问题讨论】：

尝试强制允许两个实例之间无防火墙连接的接口。例如mpirun --mca oob_tcp_if_include eth0 --mca btl_btp_if_include eth0 ...
谢谢@GillesGouaillardet。我尝试了$(which mpirun) -n 2 -hostfile $HOSTFILE_PATH/hosts2 -mca oob_tcp_if_include ens3 --mca btl_btp_if_include ens3 $EXEC_PATH/test，但它不起作用:(。我发现默认网卡是ens3而不是eth0 via ifconfig，所以我改用ens3。对吗？
听起来不错，问题可能是您的主机之间的防火墙。 sudo iptables -L 不应显示除默认 ACCEPT 策略以外的任何规则
使用sudo iptables -F后，sudo iptables -L会得到Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain InstanceServices (0 references) target prot opt source destination
这意味着机器上没有运行防火墙。 Oracle 是否通过网络防火墙限制实例之间的流量？在节点n0 上，尝试运行echo hello | nc -l 1234 和在节点n1 上，运行nc <ip of n0> 1234 < /dev/null（预期输出为hello）以确认实例之间的流量不受限制。

标签： linux ubuntu mpi oracle-cloud-infrastructure

【解决方案1】：

即使虚拟机位于同一子网中，您仍然必须允许它们之间的流量。

所以在你正在使用的子网的安全列表中打开所需的端口（@987654321@）

如果您不知道需要的端口，您可以打开所有端口（这对于生产环境来说不是一个好的做法）。

【讨论】：

谢谢。这实际上是一个与 Oracle Cloud 相关的问题。我在默认安全列表中添加了一个入口规则，它起作用了。

【解决方案2】：

有时 ufw 命令不会改变 OCI 中的 iptable。我建议您改用 iptable 命令。更多命令请参考linux-iptables-firewall-rules-examples-commands。

请使用以下命令列出所有 IPv4 规则：

sudo iptables -S

【讨论】：

感谢@Anupam Sinha。我尝试了sudo systemctl start iptables 和sudo iptables -F，然后尝试运行我的 MPI 程序，但它不起作用:(
请检查所需的防火墙端口是否在操作系统级别打开。如果不允许，则使用 iptable 命令允许它。