【问题标题】：How to connect Redis cluster from application in Kubernetes cluster?如何从 Kubernetes 集群中的应用程序连接 Redis 集群？
【发布时间】：2018-07-12 15:38:02
【问题描述】：

这是我的应用程序代码：

from flask import Flask
from redis import Redis, RedisError
import os
import socket

# Connect to Redis
redis = Redis(host=os.getenv("REDIS", "redis"), db=0, socket_connect_timeout=2, socket_timeout=2)

app = Flask(__name__)

@app.route("/")
def hello():
    try:
        visits = redis.incr("counter")
    except RedisError:
        visits = "<i>cannot connect to Redis, counter disabled</i>"

    html = "<h3>Hello {name}!</h3>" \
           "<b>Hostname:</b> {hostname}<br/>" \
           "<b>Visits:</b> {visits}"
    return html.format(name=os.getenv("NAME", "world"), hostname=socket.gethostname(), visits=visits)

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=80)

这里要连接Kubernetes集群中的一个Redis主机。所以在 Kubernetes 的 manifest 文件中放一个环境变量 REDIS 来设置值。

这是 Kubernetes 部署清单文件：

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "name" . }}
    chart: {{ template "chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ template "name" . }}
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ template "name" . }}
        release: {{ .Release.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
            - name: REDIS
              value: {{ template "fullname" . }}-master-svc
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          resources:
{{ toYaml .Values.resources | indent 12 }}
    {{- with .Values.nodeSelector }}
      nodeSelector:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.affinity }}
      affinity:
{{ toYaml . | indent 8 }}
    {{- end }}
    {{- with .Values.tolerations }}
      tolerations:
{{ toYaml . | indent 8 }}
    {{- end }}

为了连接同一个集群的Redis主机，环境值设置为：

      env:
        - name: REDIS
          value: {{ template "fullname" . }}-master-svc

关于Redis集群，官方redis-hachart安装。它的主服务清单文件是：

apiVersion: v1
kind: Service
metadata:
  name: {{ template "fullname" . }}-master-svc
  annotations:
{{ toYaml .Values.servers.annotations | indent 4 }}
spec:
  ports:
  - port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: "redis-ha"
    redis-node: "true"
    redis-role: "master"
    release: "{{ .Release.Name }}"
  type: "{{ .Values.servers.serviceType }}"

但是好像是应用程序pod没有成功连接到Redis主服务名。当我获得访问 URL 时：

export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wishful-rabbit-mychart)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

从浏览器，得到这个结果：

Hello World!

Hostname: wishful-rabbit-mychart-85dc7658c6-9blg5
Visits: cannot connect to Redis, counter disabled

更多关于pods和services的信息：

kubectl get po
NAME                                               READY     STATUS    RESTARTS   AGE
wishful-rabbit-mychart-bdfdf6558-8fjmb             1/1       Running   0          8m
wishful-rabbit-mychart-bdfdf6558-9khfs             1/1       Running   0          7m
wishful-rabbit-mychart-bdfdf6558-hgqxv             1/1       Running   0          8m
wishful-rabbit-mychart-sentinel-8667dd57d4-9njwq   1/1       Running   0          37m
wishful-rabbit-mychart-sentinel-8667dd57d4-jsq6d   1/1       Running   0          37m
wishful-rabbit-mychart-sentinel-8667dd57d4-ndqss   1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-2fn4s     1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-bwgrq     1/1       Running   0          37m
wishful-rabbit-mychart-server-746f47dfdd-spkkm     1/1       Running   0          37m

kubectl get svc
NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes                          ClusterIP   10.96.0.1        <none>        443/TCP        3h
wishful-rabbit-mychart              NodePort    10.101.103.224   <none>        80:30033/TCP   37m
wishful-rabbit-mychart-master-svc   ClusterIP   10.108.128.167   <none>        6379/TCP       37m
wishful-rabbit-mychart-sentinel     ClusterIP   10.107.63.208    <none>        26379/TCP      37m
wishful-rabbit-mychart-slave-svc    ClusterIP   10.99.211.111    <none>        6379/TCP       37m

正确的用法是什么？

Redis 服务器 Pod 环境

kubectl exec -it wishful-rabbit-mychart-server-746f47dfdd-2fn4s -- sh
/ # printenv
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
HOSTNAME=wishful-rabbit-mychart-server-746f47dfdd-2fn4s
SHLVL=1
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
HOME=/root
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
REDIS_CHART_PREFIX=wishful-rabbit-mychart-
TERM=xterm
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
PWD=/
KUBERNETES_SERVICE_HOST=10.96.0.1
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
REDIS_SENTINEL_SERVICE_HOST=redis-sentinel
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379

应用程序 Pod 环境

kubectl exec -it wishful-rabbit-mychart-85dc7658c6-8wlq6 -- sh
# printenv
KUBERNETES_SERVICE_PORT=443
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PORT=6379
KUBERNETES_PORT=tcp://10.96.0.1:443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_ADDR=10.108.128.167
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_PROTO=tcp
WISHFUL_RABBIT_MYCHART_SERVICE_HOST=10.101.103.224
HOSTNAME=wishful-rabbit-mychart-85dc7658c6-8wlq6
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP_PROTO=tcp
PYTHON_PIP_VERSION=9.0.1
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_HOST=10.107.63.208
HOME=/root
GPG_KEY=C01E1CAD5EA2C4F0B8E3571504C367C218ADD4FF
REDIS=wishful-rabbit-mychart-master-svc
WISHFUL_RABBIT_MYCHART_SERVICE_PORT=80
WISHFUL_RABBIT_MYCHART_PORT=tcp://10.101.103.224:80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT_6379_TCP=tcp://10.108.128.167:6379
WISHFUL_RABBIT_MYCHART_SENTINEL_SERVICE_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_ADDR=10.101.103.224
NAME=World
TERM=xterm
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PORT=80
WISHFUL_RABBIT_MYCHART_PORT_80_TCP_PROTO=tcp
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
LANG=C.UTF-8
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_ADDR=10.107.63.208
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_HOST=10.99.211.111
WISHFUL_RABBIT_MYCHART_PORT_80_TCP=tcp://10.101.103.224:80
PYTHON_VERSION=2.7.14
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PORT=26379
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_SERVICE_PORT_HTTPS=443
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_HOST=10.108.128.167
KUBERNETES_SERVICE_HOST=10.96.0.1
PWD=/app
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT=tcp://10.99.211.111:6379
WISHFUL_RABBIT_MYCHART_SERVICE_PORT_HTTP=80
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_SLAVE_SVC_PORT_6379_TCP_ADDR=10.99.211.111
WISHFUL_RABBIT_MYCHART_SENTINEL_PORT_26379_TCP=tcp://10.107.63.208:26379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_SERVICE_PORT=6379
WISHFUL_RABBIT_MYCHART_MASTER_SVC_PORT=tcp://10.108.128.167:6379

所有端点

kubectl get ep
NAME                                ENDPOINTS                                              AGE
kubernetes                          10.0.2.15:8443                                         4h
wishful-rabbit-mychart              172.17.0.5:80,172.17.0.6:80,172.17.0.7:80              1h
wishful-rabbit-mychart-master-svc   <none>                                                 1h
wishful-rabbit-mychart-sentinel     172.17.0.11:26379,172.17.0.12:26379,172.17.0.8:26379   1h
wishful-rabbit-mychart-slave-svc    <none>

描述 Redis 主服务

kubectl describe svc wishful-rabbit-mychart-master-svc
Name:              wishful-rabbit-mychart-master-svc
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=redis-ha,redis-node=true,redis-role=master,release=wishful-rabbit
Type:              ClusterIP
IP:                10.108.128.167
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Endpoints:         <none>
Session Affinity:  None
Events:            <none>

描述 Redis 服务器 Pod

kubectl describe po wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Name:           wishful-rabbit-mychart-server-746f47dfdd-2fn4s
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Fri, 02 Feb 2018 15:28:42 +0900
Labels:         app=mychart
                chart=mychart-0.1.0
                heritage=Tiller
                name=redis-server
                pod-template-hash=3029038988
                podIP=172.17.0.10
                redis-node=true
                redis-role=master
                release=wishful-rabbit
                runID=cbf8e0
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wishful-rabbit-mychart-server-746f47dfdd","uid":"4fcb0dfc-07e2-1...
Status:         Running
IP:             172.17.0.10
Controlled By:  ReplicaSet/wishful-rabbit-mychart-server-746f47dfdd
Containers:
  redis:
    Container ID:   docker://2734d60bd44a1446ec6556369359ed15b866a4589abe1e6d394f9252114c6d4d
    Image:          quay.io/smile/redis:4.0.6r2
    Image ID:       docker-pullable://quay.io/smile/redis@sha256:8948a952920d4495859c984546838d4c9b4c71e0036eef86570922d91cacb3df
    Port:           6379/TCP
    State:          Running
      Started:      Fri, 02 Feb 2018 15:28:44 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      REDIS_SENTINEL_SERVICE_HOST:  redis-sentinel
      REDIS_CHART_PREFIX:           wishful-rabbit-mychart-
    Mounts:
      /redis-master-data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wfv2q (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-wfv2q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wfv2q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:          <none>

【问题讨论】：

您共享了清单文件模板，而不是实际的清单文件。这些对于帮助您解决问题并不是很有用。我建议使用 kubectl exec 进入您的应用程序 pod 并检查 REDIS env var 的实际值，然后尝试从那里连接以了解有关该问题的更多信息。
@whites11 我认为问题的关键在于如何从环境变量中连接redis-ha 集群。我检查了Redis Master pod 环境变量。得到HOSTNAME=wishful-rabbit-mychart-server-746f47dfdd-2fn4s 的主机名。不能使用它，因为746f47dfdd-2fn4s 是动态变化的。 REDIS_SENTINEL_SERVICE_HOST=redis-sentinel 用于连接哨兵。仍然不清楚要连接哪个 ENV。
您需要检查应用程序 pod 的 env var，而不是 redis 的。
另外加上$ kubectl get endpoints
@whites11 在上述问题中添加了应用程序 pod env。它有REDIS=wishful-rabbit-mychart-master-svc，但无法访问wishful-rabbit-mychart-master-svc 服务。

标签： redis kubernetes cluster-computing minikube kubernetes-helm

【解决方案1】：

您的wishful-rabbit-mychart-master-svc 服务的Selector 与您的主redis pod 的labels 之间存在差异。

您的服务正在搜索具有以下标签的 pod：

app=redis-ha
redis-node=true
redis-role=master
release=wishful-rabbit

虽然您的 pod 具有以下标签：

app=mychart
chart=mychart-0.1.0
heritage=Tiller
name=redis-server
pod-template-hash=3029038988
podIP=172.17.0.10
redis-node=true
redis-role=master
release=wishful-rabbit
runID=cbf8e0

如您所见，app 标签不同（redis-ha 在您的服务中，mychart 在您的 pod 中）。

这会导致服务不受限制（它不知道将传入流量转发到哪里）。

虽然我不知道此配置错误的实际原因，但我可以建议一个解决方案以使其正常工作。

您必须编辑您的 redis 服务并更改其选择器属性以匹配 pod 的标签。运行：

kubectl edit svc wishful-rabbit-mychart-master-svc

并将app: "redis-ha" 更改为app: "mychart"。

您的应用程序应该突然能够访问您的 redis 实例。

【讨论】：

你是对的！非常感谢您的正确回答！我会告诉你为什么会发生这个问题。我正在使用Helm 将我的应用程序和redis-ha 上下文包含在同一个图表中。在我的应用程序中，我将它的 Selector 命名为 mychart。然后将redis-ha文件复制到默认Selectorredis-ha的同一个目录，所以它们不匹配！ github.com/kubernetes/charts/blob/master/stable/redis-ha/…

【解决方案2】：

Kubernetes Service 是一种抽象，它定义了一组逻辑 Pod。 Service 所针对的 Pod 集合（通常）由标签选择器确定

因此，Pod 可以在选择器的帮助下被 Service 定位。

Service 的选择器将被持续评估，结果将被 POST 到 Endpoints 对象。

如果在您提供的选择器中找不到 Pod，则不会创建有效的 Endpoint。

在这种情况下，如果你使用这个 Service 来访问 Pod，这是行不通的，因为 DNS 不会从 Endpoint 获取任何 IP 地址

要了解更多信息，read how Service works

【讨论】：

检查这个。这将帮助您正确理解场景。
非常感谢您的帮助！我现在找到了关键原因！你和@whites11 给了我一个非常正确的解决方案和解释！