TLS 不适用于 Kubernetes 中的负载均衡器后端服务答案

【问题标题】：TLS doesn't work with LoadBalancer backed Service in KubernetesTLS 不适用于 Kubernetes 中的负载均衡器后端服务
【发布时间】：2019-12-14 08:57:48
【问题描述】：

我正在尝试通过将服务类型创建为负载平衡器来公开集群中的应用程序。这样做的原因是我希望这个应用程序有一个单独的沟通渠道。我有一个 KOPS 集群。我想使用 AWS 的网络负载均衡器，以便它获得静态 IP。当我创建服务时，端口 80 映射到应用程序正在运行的端口，一切正常，但是当我尝试添加端口 443 时，它只是超时。

这是有效的配置 -

apiVersion: v1
metadata:
  name: abc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  labels:
    app: abc
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 9050
  selector:
    app: abc
  type: LoadBalancer

只要我在配置文件中添加 TLS 支持并部署它。与负载均衡器的连接超时。如何向负载均衡器添加 TLS 支持？我想通过服务而不是通过入口来做到这一点。这是对我不起作用的配置，当我将链接粘贴到浏览器中时，它会超时。

kind: Service
apiVersion: v1
metadata:
  name: abc
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: xxxxx
  labels:
    app: abc
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 443
    protocol: TCP
    targetPort: 9050
  selector:
    app: abc
  type: LoadBalancer

【问题讨论】：

标签： amazon-web-services kubernetes kubectl kubernetes-ingress kops

【解决方案1】：

您的注释是指https 端口

service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

但你的端口名为http，改为https

spec:
  externalTrafficPolicy: Local
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 9050

【讨论】：

【解决方案2】：

您现在可以使用 NLB 和 SSL 终止部署入口（NLB 中的 https > 服务中的 http）。终于找到了适合我的解决方案，您可以尝试部署以下 ingress.yaml（确保在部署部分下更新您的证书 ARN）：

    ---
# Source: nginx-ingress/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
---
# Source: nginx-ingress/templates/default-backend-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-backend
---
# Source: nginx-ingress/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "controller"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
data:
  server-snippet: |
    listen 8000;
    if ( $server_port = 80 ) {
       return 308 https://$host$request_uri;
    }
  ssl-redirect: "false"
---
# Source: nginx-ingress/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
---
# Source: nginx-ingress/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress
subjects:
  - kind: ServiceAccount
    name: nginx-ingress
    namespace: ingress-nginx
---
# Source: nginx-ingress/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
rules:
  - apiGroups:
      - ""
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - "networking.k8s.io" # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - create
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: nginx-ingress/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress
subjects:
  - kind: ServiceAccount
    name: nginx-ingress
    namespace: ingress-nginx
---
# Source: nginx-ingress/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:##REPLACE WITH YOUR CERT ARN"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "controller"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-controller
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: special
  selector:
    app: nginx-ingress
    release: nginx-ingress
    app.kubernetes.io/component: controller
  type: "LoadBalancer"
---
# Source: nginx-ingress/templates/default-backend-service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    component: "default-backend"
    heritage: Helm
    release: nginx-ingress
  name: nginx-ingress-default-backend
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
  selector:
    app: nginx-ingress
    release: nginx-ingress
    app.kubernetes.io/component: default-backend
  type: "ClusterIP"
---
# Source: nginx-ingress/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
    app.kubernetes.io/component: controller
  name: nginx-ingress-controller
  annotations:
    {}
spec:
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  replicas: 1
  revisionHistoryLimit: 10
  strategy:
    {}
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app: nginx-ingress
        release: nginx-ingress
        component: "controller"
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      containers:
        - name: nginx-ingress-controller
          image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/nginx-ingress-default-backend
            - --publish-service=$(POD_NAMESPACE)/nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/nginx-ingress-controller
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            capabilities:
                drop:
                - ALL
                add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: special
              containerPort: 8000
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          resources:
            {}
      hostNetwork: false
      serviceAccountName: nginx-ingress
      terminationGracePeriodSeconds: 60
---
# Source: nginx-ingress/templates/default-backend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ingress-nginx
  labels:
    app: nginx-ingress
    chart: nginx-ingress-1.38.0
    heritage: Helm
    release: nginx-ingress
    app.kubernetes.io/component: default-backend
  name: nginx-ingress-default-backend
spec:
  selector:
    matchLabels:
      app: nginx-ingress
      release: nginx-ingress
  replicas: 1
  revisionHistoryLimit: 10
  template:
    metadata:
      labels:
        app: nginx-ingress
        release: nginx-ingress
        app.kubernetes.io/component: default-backend
    spec:
      containers:
        - name: nginx-ingress-default-backend
          image: "k8s.gcr.io/defaultbackend-amd64:1.5"
          imagePullPolicy: "IfNotPresent"
          args:
          securityContext:
            runAsUser: 65534
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            {}
      serviceAccountName: nginx-ingress-backend
      terminationGracePeriodSeconds: 60

【讨论】：

【解决方案3】：

您可以使用 tls & ssl 终止

   apiVersion: v1
    kind: Service
    metadata:
      name: test-service
      annotations:
        # Note that the backend talks over HTTP.
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
        # TODO: Fill in with the ARN of your certificate.
        service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:{region}:{user id}:certificate/{id}
        # Only run SSL on the port named "https" below.
        service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    spec:
      selector:
        app: test-pod
      ports:
      - name: http
        port: 80
        targetPort: 8080
      - name: https
        port: 443
        targetPort: 8080
      type: LoadBalancer

您可以在 aws 证书管理器中添加 tls 证书，并将证书的 arn 地址用于 kubernetes 服务。

就像在 becked 中一样，您可以终止 https 连接并仅使用 HTTP。

你也可以看看这个：https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/

https://github.com/kubernetes/kubernetes/issues/73297

编辑：1

service.beta.kubernetes.io/aws-load-balancer-type: nlb

如果不起作用，请尝试根据您的负载均衡器类型添加此注释。

【讨论】：

这就是我的问题。我已经添加了添加 SSL 终止所需的所有配置，但是当我添加 TLS 支持时，我无法访问该应用程序。 TLS 配置对我不起作用。
@AnshulTripathi 您正在使用哪个 kubernetes 版本？在 Kubernetes 1.8 中，如果不自定义 nginx 配置模板，您目前无法在 ELB 中终止 TLS。
@AnshulTripathi 检查编辑 1 我已经更新了答案。
我的 Kube 版本是 v1.10.12。
我已经有了配置 - service.beta.kubernetes.io/aws-load-balancer-type: "nlb"