我的 pod 无法连接到私有子网答案

【问题标题】：My pod can't connect to the private subnet我的 pod 无法连接到私有子网
【发布时间】：2020-06-27 07:04:42
【问题描述】：

[Interface:vetha13c9067] 05:57:57.851421 IP 10.28.0.7 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:cbr0] 05:57:57.851421 IP 10.28.0.7 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:eth0] 05:57:57.851614 IP 172.18.0.25 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64

以上日志没有任何问题10.28.0.7可以与172.28.10.17成功通信

[Interface:veth916b4093] 05:57:09.699334 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:cbr0] 05:57:09.699334 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:eth0] 05:57:09.699380 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64

上面的日志有些麻烦，你可以看到它的eth0接口请求仍然来自10.20.4.194而不是172.18.0.0/16，它的子网范围。

为什么我的 pod 中的请求不是来自子网范围（172.18.0.0/16）？

以下是我当前的配置：

bash-5.0# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if135: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default
    link/ether 76:f9:ea:bf:d7:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.4.194/24 scope global eth0
       valid_lft forever preferred_lft forever
bash-5.0# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.20.4.1       0.0.0.0         UG        0 0          0 eth0
10.20.4.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
bash-5.0# ip neigh sh
10.20.4.1 dev eth0 lladdr 92:7f:08:52:f9:d4 STALE
bash-5.0# cat /etc/resolv.conf
nameserver 10.85.0.10
search default.svc.cluster.local svc.cluster.local cluster.local c.buzzdata.internal google.internal
options ndots:5
bash-5.0# nslookup kubernetes
Server:     10.85.0.10
Address:    10.85.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.85.0.1

╰─ k describe po netshoot-container
Name:               netshoot-container
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-search-cluster-pool-765be39a-gkt4/172.18.0.17
Start Time:         Mon, 16 Mar 2020 14:05:32 +0900
Labels:             run=netshoot-container
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container netshoot-container
Status:             Running
IP:                 10.20.4.194
Containers:
  netshoot-container:
    Container ID:  docker://0df9d4b262f926d7d89a42f58de672284a5cb5637ab951b752e0c8b34ded676a
    Image:         nicolaka/netshoot
    Image ID:      docker-pullable://nicolaka/netshoot@sha256:99d15e34efe1e3c791b0898e05be676084638811b1403fae59120da4109368d4
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/bash
    State:          Running
      Started:      Mon, 16 Mar 2020 14:05:36 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xxxx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xxxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xxxx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/hostname=gke-xxxx-cluster-pool-xxxx-gkt4
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Chain INPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  10.20.0.0/14         anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.20.0.0/14         /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination

【问题讨论】：

前两个日志用tcpdump -i any host 172.28.10.17记录
很抱歉，您的帖子让我有些困惑，您介意再澄清一下吗？你的环境是什么？您提到了 docker，但是您的输出，kubernetes 和 GKE，澄清了它，最后是什么问题以及预期的行为是什么？我还建议阅读about page 并访问这些链接以了解how to ask a good question 和how to create a minimal, reproducible example
感谢@willrof 的建议，抱歉我的问题很脆弱，我很着急 D：实际上 docker 不是问题，因为 docker 制造的容器可以毫无问题地 ping 172.18.0.1，但是容器k8s做的不能ping通172.18.0.1

标签： docker networking kubernetes containers vpn

【解决方案1】：

我找到了原因，那是因为启用了 IP-MASQ。

如果您遇到 pod 无法与私有子网（如 172.x.x.x）通信的问题，请通过 iptables -t nat -L 检查您节点的 iptable 规则

如果你能看到 IP-MASQ 规则并且如果有 RETURN all -- anywhere 172.16.0.0/12 /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */ 然后下面出来！

https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

手动修改 iptable 可能是其中一种解决方案，但我认为最好遵循文档。

【讨论】：