K8s ingress：nginx入口控制器未处于运行模式

Question

我有一个 jenkins 图像，我作为 NodeType 提供服务。它运作良好。由于我会添加更多的服务，所以我需要使用 ingress nginx 将流量分流到不同类型的服务。

此时，我用我的win10设置了两个vm（Centos 7.5）。一个 vm 作为 master1，它有两个内部 IPv4 地址 (10.0.2.9 and 192.168.56.103) 和一个 vm 作为 worker node4 (10.0.2.6 and 192.168.56.104)。

所有图片都是本地的。我已下载到本地 docker 映像存储库。问题是 Nginx 入口没有运行。

我的配置如下：

ingress-nginx-ctl.yaml:

apiVersion: extensions/v1beta1
metadata:
  name: ingress-nginx
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
        name: ingress-nginx
        imagePullPolicy: Never
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend

ingress-nginx-res.yaml：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
spec:
  rules:
  - host:
    http:
      paths:
      - path: /
        backend:
          serviceName: shinyinfo-jenkins-svc
          servicePort: 8080

nginx-default-backend.yaml

kind: Service
apiVersion: v1
metadata:
  name: nginx-default-backend
  namespace: default
spec:
  ports:
  - port: 80
    targetPort: http
  selector:
    app: nginx-default-backend
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: nginx-default-backend
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-default-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        image: chenliujin/defaultbackend
        imagePullPolicy: Never
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 10m
            memory: 10Mi
          requests:
            cpu: 10m
            memory: 10Mi
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

shinyinfo-jenkins-pod.yaml

apiVersion: v1
kind: Pod
metadata:
 name: shinyinfo-jenkins
 labels:
   app: shinyinfo-jenkins
spec:
 containers:
   - name: shinyinfo-jenkins
     image: shinyinfo_jenkins
     imagePullPolicy: Never
     ports:
       - containerPort: 8080
         containerPort: 50000
     volumeMounts:
     - mountPath: /devops/password
       name: jenkins-password
     - mountPath: /var/jenkins_home
       name: jenkins-home
 volumes:
   - name: jenkins-password
     hostPath:
       path: /jenkins/password
   - name: jenkins-home
     hostPath:
       path: /jenkins

shinyinfo-jenkins-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: shinyinfo-jenkins-svc
  labels:
    name: shinyinfo-jenkins-svc
spec:
  selector:
    app: shinyinfo-jenkins
  type: NodePort
  ports:
  - name: tcp
    port: 8080
    nodePort: 30003

nginx ingress有问题，控制台输出如下：

[master@master1 config]$ sudo kubectl apply -f ingress-nginx-ctl.yaml
service/ingress-nginx created
deployment.extensions/ingress-nginx created

[master@master1 config]$ sudo kubectl apply -f ingress-nginx-res.yaml
ingress.extensions/my-ingress created

图像是 CrashLoopBackOff，为什么？？？

[master@master1 config]$ sudo kubectl get po
NAME                                     READY     STATUS             RESTARTS   AGE
ingress-nginx-66df6b6d9-mhmj9            0/1       CrashLoopBackOff   1          9s
nginx-default-backend-645546c46f-x7s84   1/1       Running            0          6m
shinyinfo-jenkins                        1/1       Running            0          20m

描述吊舱：

[master@master1 config]$ sudo kubectl describe po ingress-nginx-66df6b6d9-mhmj9
Name:               ingress-nginx-66df6b6d9-mhmj9
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               node4/192.168.56.104
Start Time:         Thu, 08 Nov 2018 16:45:46 +0800
Labels:             app=ingress-nginx
                    pod-template-hash=228926285
Annotations:        <none>
Status:             Running
IP:                 100.127.10.211
Controlled By:      ReplicaSet/ingress-nginx-66df6b6d9
Containers:
  ingress-nginx:
    Container ID:  docker://2aba164d116758585abef9d893a5fa0f0c5e23c04a13466263ce357ebe10cb0a
    Image:         quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
    Image ID:      docker://sha256:a3f21ec4bd119e7e17c8c8b2bf8a3b9e42a8607455826cd1fa0b5461045d2fa9
    Ports:         80/TCP, 443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Thu, 08 Nov 2018 16:46:09 +0800
      Finished:     Thu, 08 Nov 2018 16:46:09 +0800
    Ready:          False
    Restart Count:  2
    Liveness:       http-get http://:10254/healthz delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-66df6b6d9-mhmj9 (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-24hnm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-24hnm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-24hnm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  40s                default-scheduler  Successfully assigned default/ingress-nginx-66df6b6d9-mhmj9 to node4
  Normal   Pulled     18s (x3 over 39s)  kubelet, node4     Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0" already present on machine
  Normal   Created    18s (x3 over 39s)  kubelet, node4     Created container
  Normal   Started    17s (x3 over 39s)  kubelet, node4     Started container
  Warning  BackOff    11s (x5 over 36s)  kubelet, node4     Back-off restarting failed container

pod 日志：

[master@master1 config]$ sudo kubectl logs ingress-nginx-66df6b6d9-mhmj9
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.20.0
  Build:      git-e8d8103
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------
nginx version: nginx/1.15.5
W1108 08:47:16.081042       6 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1108 08:47:16.081234       6 main.go:196] Creating API client for https://10.96.0.1:443
I1108 08:47:16.122315       6 main.go:240] Running in Kubernetes cluster version v1.11 (v1.11.3) - git (clean) commit a4529464e4629c21224b3d52edfe0ea91b072862 - platform linux/amd64
F1108 08:47:16.123661       6 main.go:97] ✖ The cluster seems to be running with a restrictive Authorization mode and the Ingress controller does not have the required permissions to operate normally.

这里的专家可以给我一些提示吗？

Answer 1

您需要设置 ingress-nginx 以使用单独的服务帐户并为服务帐户提供必要的权限。

这是一个例子：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lb
  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-normal
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - "ingress-controller-leader-dev"
      - "ingress-controller-leader-prod"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-minimal
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-minimal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-normal
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-normal
subjects:
  - kind: ServiceAccount
    name: lb
    namespace: kube-system