如何在裸机上使用 hostNetwork 安装 nginx-ingress？答案

【问题标题】：How to install nginx-ingress with hostNetwork on bare-metal?如何在裸机上使用 hostNetwork 安装 nginx-ingress？
【发布时间】：2019-11-16 19:31:53
【问题描述】：

我在裸机上创建了一个单节点 kubeadm 集群，经过一些研究后，我会选择主机网络方法 (https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network)，因为由于网络限制，NodePort 不是一个选项。

我尝试通过命令安装带有 helm chart 的 nginx-ingress：

   helm install stable/nginx-ingress \
     --set controller.hostNetwork=true

问题是它正在创建一个永远挂起的 LoadBalancer 服务，并且我的入口对象没有被路由：

NAME                                                                 READY   STATUS    RESTARTS   AGE
pod/whopping-kitten-nginx-ingress-controller-5db858b48c-dp2j8        1/1     Running   0          5m34s
pod/whopping-kitten-nginx-ingress-default-backend-5c574f4449-dr4xm   1/1     Running   0          5m34s

NAME                                                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/kubernetes                                      ClusterIP      10.96.0.1       <none>        443/TCP                      6m43s
service/whopping-kitten-nginx-ingress-controller        LoadBalancer   10.97.143.40    <pending>     80:30068/TCP,443:30663/TCP   5m34s
service/whopping-kitten-nginx-ingress-default-backend   ClusterIP      10.106.217.96   <none>        80/TCP                       5m34s

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/whopping-kitten-nginx-ingress-controller        1/1     1            1           5m34s
deployment.apps/whopping-kitten-nginx-ingress-default-backend   1/1     1            1           5m34s

NAME                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/whopping-kitten-nginx-ingress-controller-5db858b48c        1         1         1       5m34s
replicaset.apps/whopping-kitten-nginx-ingress-default-backend-5c574f4449   1         1         1       5m34s

是否需要进行任何其他配置才能在此方法中取得成功？

更新：这里是 ingress-controller pod 的日志

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0707 19:02:50.552631       6 flags.go:185] Watching for Ingress class: nginx
W0707 19:02:50.552882       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0707 19:02:50.556215       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0707 19:02:50.556368       6 main.go:205] Creating API client for https://10.96.0.1:443
I0707 19:02:50.562296       6 main.go:249] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0707 19:02:51.357524       6 main.go:102] Validated default/precise-bunny-nginx-ingress-default-backend as the default backend.
I0707 19:02:51.832384       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0707 19:02:53.516654       6 store.go:613] Unexpected error reading configuration configmap: configmaps "precise-bunny-nginx-ingress-controller" not found
I0707 19:02:53.527297       6 nginx.go:265] Starting NGINX Ingress controller
I0707 19:02:54.630002       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"801", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/staging-ingress
I0707 19:02:54.727989       6 nginx.go:311] Starting NGINX process
I0707 19:02:54.728249       6 leaderelection.go:217] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
W0707 19:02:54.729235       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:54.729334       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:54.729442       6 controller.go:797] Service "default/test" does not have any active Endpoint.
I0707 19:02:54.729535       6 controller.go:170] Configuration changes detected, backend reload required.
I0707 19:02:54.891620       6 controller.go:188] Backend successfully reloaded.
I0707 19:02:54.891654       6 controller.go:202] Initial sync, sleeping for 1 second.
I0707 19:02:54.948639       6 leaderelection.go:227] successfully acquired lease default/ingress-controller-leader-nginx
I0707 19:02:54.949148       6 status.go:86] new leader elected: precise-bunny-nginx-ingress-controller-679b9557ff-n57mc
[07/Jul/2019:19:02:55 +0000]TCP200000.000
W0707 19:02:58.062645       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:58.062676       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:58.062686       6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:02.406151       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:03:02.406188       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:02.406357       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:02 +0000]TCP200000.000
W0707 19:03:05.739438       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:05.739467       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:05 +0000]TCP200000.001
W0707 19:03:09.072793       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:09.072820       6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:12.406121       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:12.406143       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:15 +0000]TCP200000.000
I0707 19:03:54.959607       6 status.go:295] updating Ingress default/staging-ingress status from [] to [{ }]
I0707 19:03:54.961925       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"1033", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/staging-ingress

【问题讨论】：

标签： nginx kubernetes kubernetes-ingress bare-metal-server

【解决方案1】：

@ijaz-ahmad-khan @vkr 给出了解决问题的好主意，但完整的设置步骤如下：

1) 使用以下命令安装 nginx-ingress：

helm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet

2) 在您的部署中放置：

spec:
  template:
    spec:
      hostNetwork: true

3) 在所有 Ingress 对象中放置：

metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"

【讨论】：

您是如何安装 helm chart 的？为controller.service.type 指定一个空值会导致Error: release nginx-ingress failed: Service "nginx-ingress-controller" is invalid: spec.type: Unsupported value: "''": supported values: "ClusterIP", "ExternalName", "LoadBalancer", "NodePort"。
我猜第 2 步不是必需的，因为 controller.hostNetwork 你已经在设置它 github.com/helm/charts/blob/master/stable/nginx-ingress/…
@papanito 在我遇到这个问题时，我确信我没有尝试过。也许现在有了最新版本的 kubernetes 和 nginx-ingress，就不再需要了。你能试试吗？我也可以确认并更新答案。
可能，只是想提一下；-)
@KarlRichter 可能取决于您正在运行的 Kubernetes 的确切版本。裸机安装文档建议只删除 Ingress Controller 服务，因为它不再与 DaemonSet/hostNetwork 设置一起使用。因此，您可以使用任何合法值来让安装正常工作并在之后删除服务。

【解决方案2】：

我会说您需要与Ingress on Custom Kubernetes 文章中提供的配置接近的配置。并同意@Ijaz Ahmad Khan - 应该是kind: DaemonSet

例子：

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
            hostPort: 80
          - name: https
            containerPort: 443
            hostPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          securityContext:
            runAsNonRoot: false

【讨论】：

鉴于您的回答和@IjazAhmadKhan，我在 values.yaml 中发现了一些似乎是建议的解决方案的属性：helm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet 我没有验证它是否按预期工作。
此配置的行为与部署类型的控制器完全相同，并且不起作用。任何应该响应的网址都会给我一个超时。

【解决方案3】：

在某些教程中，您可能会被建议简单地做一个 kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml

但如果您在没有负载平衡器的私有 LAN 上使用裸机实现，则需要稍微修改此步骤。

不要直接应用 deploy.yaml，而是执行 wget（例如 wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml）然后编辑 deploy.yaml - 向下滚动到 Deployment 资源并添加 hostNetwork: true 键：值对。比如：

      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      hostNetwork: true
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission

等等

所以你正在做的是修改控制器部署的模板规范。

使用以下内容部署修改后的版本：

kubectl apply -f deploy.yaml

您为使用此部署/控制器而创建的 Ingress 资源应像这样引用它：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-api-ingress
  annotations:
    kubernetes.io/ingress.class: nginx

然后你可以做这样的描述：

kubectl describe service -n ingress-nginx ingress-nginx-controller

并找出哪个幸运节点被指定为您的入口。

【讨论】：