【问题标题】:502 Bad Gateway issue with some containers某些容器的 502 Bad Gateway 问题
【发布时间】:2020-06-26 02:55:39
【问题描述】:

我在集群中安装了NGINX Ingress,但是当我尝试访问它时,我得到了 502

这些是我安装 NGINX Ingress 所遵循的步骤

kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f daemon-set/nginx-ingress.yaml

这是我试图从入口控制器访问的部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ngtest
  namespace: practice
  labels: 
    app: ngtest
spec:
  replicas: 1
  selector:
    matchLabels: 
      app: ngtest
  template: 
    metadata:
      labels: 
        app: ngtest
    spec: 
      imagePullSecrets:
      - name: testkuldeepsecret
      containers:
      - name: ngtest
        image: nginx:latest
        ports:
        - containerPort: 80


---
apiVersion: v1
kind: Service
metadata:                     
  name: ngtest
  namespace: practice
  labels: 
spec:
  ports: 
  - port: 80 
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: ngtest

使用集群 IP 我可以访问 nginx

现在我配置了 Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: practice-ingress
  namespace: practice
spec:
  rules:
  - host: practice.example.com
    http: 
      paths:
      - path: /
        backend:
          serviceName: ngtest
          servicePort: 80

入口描述

Name:             practice-ingress
Namespace:        practice
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  practice.example.com
                        /   ngtest:80 (10.32.0.7:80)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"practice-ingress","namespace":"practice"},"spec":{"rules":[{"host":"practice.example.com","http":{"paths":[{"backend":{"serviceName":"ngtest","servicePort":80},"path":"/"}]}}]}}

Events:
  Type    Reason          Age   From                      Message
  ----    ------          ----  ----                      -------
  Normal  AddedOrUpdated  46m   nginx-ingress-controller  Configuration for practice/practice-ingress was added or updated
  Normal  AddedOrUpdated  46m   nginx-ingress-controller  Configuration for practice/practice-ingress was added or updated

很明显,ingress 会将其转发到 80 端口,并且 ngtest 服务也暴露在 80 端口上

我在主机文件中添加了 IP 地址和 practice.example.com。

所以当我尝试访问 http://practice.example.com 时,我得到 502

入口 NGINX Pod 日志

10.32.0.1 - - [14/Mar/2020:07:37:53 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:56 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
2020/03/14 07:37:56 [error] 30#30: *15 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:59 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:37:59 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:03 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:03 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:06 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:06 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:09 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:09 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:12 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:12 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:07 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /practice HTTP/1.1", upstream: "http://10.32.0.7:80/practice", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:52:07 +0000] "GET /practice HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:11 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:52:11 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:48 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:48 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:58 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:58 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:36 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:08:13:36 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:39 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:08:13:39 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"

正在运行的资源

root@ip-172-31-12-0:/home/ubuntu# kubectl get all --all-namespaces
NAMESPACE       NAME                                         READY   STATUS    RESTARTS   AGE
kube-system     pod/coredns-6955765f44-7fxkh                 1/1     Running   0          6d13h
kube-system     pod/coredns-6955765f44-l7dkf                 1/1     Running   0          6d13h
kube-system     pod/etcd-ip-172-31-12-0                      1/1     Running   0          6d13h
kube-system     pod/kube-apiserver-ip-172-31-12-0            1/1     Running   0          3h2m
kube-system     pod/kube-controller-manager-ip-172-31-12-0   1/1     Running   1          6d13h
kube-system     pod/kube-proxy-2h672                         1/1     Running   1          6d13h
kube-system     pod/kube-proxy-7zc4j                         1/1     Running   0          6d13h
kube-system     pod/kube-scheduler-ip-172-31-12-0            1/1     Running   1          6d13h
kube-system     pod/weave-net-wrrml                          2/2     Running   0          6d13h
kube-system     pod/weave-net-wxjv4                          2/2     Running   4          6d13h
nginx-ingress   pod/nginx-ingress-9hsck                      1/1     Running   0          58m
nginx-ingress   pod/nginx-ingress-xvhfg                      1/1     Running   0          58m
practice        pod/customer-855c6d9bc8-8pq8v                1/1     Running   0          14h
practice        pod/ngtest-6c5ff8c7df-2867b                  1/1     Running   0          46m
practice        pod/some-mysql-7896cfbb4c-zlpl8              1/1     Running   0          38h

NAMESPACE     NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes         ClusterIP   10.96.0.1        <none>        443/TCP                  6d13h
kube-system   service/kube-dns           ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   6d13h
practice      service/customer-service   ClusterIP   10.110.222.12    <none>        80/TCP                   14h
practice      service/ngtest             ClusterIP   10.105.115.222   <none>        80/TCP                   46m
practice      service/some-mysql         ClusterIP   None             <none>        3306/TCP                 38h

NAMESPACE       NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system     daemonset.apps/kube-proxy      2         2         2       2            2           beta.kubernetes.io/os=linux   6d13h
kube-system     daemonset.apps/weave-net       2         2         2       2            2           <none>                        6d13h
nginx-ingress   daemonset.apps/nginx-ingress   2         2         2       2            2           <none>                        58m

NAMESPACE     NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns      2/2     2            2           6d13h
practice      deployment.apps/customer     1/1     1            1           14h
practice      deployment.apps/ngtest       1/1     1            1           46m
practice      deployment.apps/some-mysql   1/1     1            1           38h

NAMESPACE     NAME                                    DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-6955765f44      2         2         2       6d13h
practice      replicaset.apps/customer-855c6d9bc8     1         1         1       14h
practice      replicaset.apps/ngtest-6c5ff8c7df       1         1         1       46m
practice      replicaset.apps/some-mysql-7896cfbb4c   1         1         1       38h

【问题讨论】:

    标签: docker nginx kubernetes nginx-ingress


    【解决方案1】:

    No route to host 表示 nginx 入口控制器无法连接到 pod IP。检查运行 nginx 入口控制器的节点和运行 pod 的节点之间是否有任何防火墙。

    【讨论】:

    • 是节点正在运行并且没有防火墙
    • 如果两个 pod 是他们的,那么它可以工作,无法理解为什么一个 pod 不能工作
    • 我认为如果 pod 与 nginx 控制器在同一个节点上运行,那么它可以正常工作
    【解决方案2】:

    是否尝试使用https://github.com/kubernetes/ingress-nginx?大多数时候我们使用了错误的入口。

    【讨论】:

      猜你喜欢
      • 1970-01-01
      • 1970-01-01
      • 2017-05-10
      • 2014-03-05
      • 1970-01-01
      • 2016-09-30
      • 1970-01-01
      • 2020-04-05
      • 2020-07-12
      相关资源
      最近更新 更多