【发布时间】:2018-02-03 13:10:08
【问题描述】:
我的集群上有一个单实例 Redis 部署/服务:
Redis.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myapp-redis
labels:
name: myapp-redis
spec:
ports:
- port: 6379
targetPort: 6379
selector:
name: myapp-redis
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-redis
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-redis
labels:
name: myapp-redis
spec:
selector:
matchLabels:
name: myapp-redis
strategy:
type: Recreate
replicas: 1
template:
metadata:
labels:
name: myapp-redis
spec:
containers:
- name: myapp-redis
image: registry/myapp-redis:0.0.0-alpha.13
imagePullPolicy: Always
ports:
- containerPort: 6379
volumeMounts:
- name: myapp-redis
mountPath: /etc/redis/
imagePullSecrets:
- name: regsecret
volumes:
- name: myapp-redis
persistentVolumeClaim:
claimName: myapp-redis
---
Redis 服务说明
我从kubectl describe svc myapp-redis -n mw-dev 得到这个:
Name: myapp-redis
Namespace: mw-dev
Labels: name=myapp-redis
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"myapp-redis"},"name":"myapp-redis","namespace":"mw-dev"},"sp...
Selector: name=myapp-redis
Type: ClusterIP
IP: 10.3.0.137
Port: <unset> 6379/TCP
TargetPort: 6379/TCP
Endpoints: 10.2.2.173:6379
Session Affinity: None
Events: <none>
检查 redis 是否启动并运行
确保数据库已启动并正在运行,我可以使用kubectl exec -it myapp-redis-[..] sh -n mw-dev 在 pod 内打开一个 shell,并使用 redis-cli -a test ping ping 数据库。如果我这样做,我会收到一个PONG,因此密码 (test) 似乎已解析并且数据库已启动。
python应用连接到redis服务的问题
但是,如果我尝试将运行 Python 应用程序的 pod 连接到 redis 数据库,我会收到来自 Python 应用程序的连接被拒绝错误。
kubectl logs myapp-backend-596... -n mw-dev
[...]
File "/usr/local/lib/python3.6/site-packages/aioredis/stream.py", line 19, in open_connection
lambda: protocol, host, port, **kwds)
File "uvloop/loop.pyx", line 1733, in create_connection
File "uvloop/loop.pyx", line 1712, in uvloop.loop.Loop.create_connection
ConnectionRefusedError: [Errno 111] Connection refused
这是 Python 应用程序的配置:
Backend.yaml
---
apiVersion: v1
kind: Service
metadata:
name: myapp-backend
labels:
name: myapp-backend
spec:
ports:
- port: 8000
targetPort: 8000
selector:
name: myapp-backend
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-backend
labels:
name: myapp-backend
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
name: myapp-backend
spec:
containers:
- name: myapp-backend
image: registry/myapp-backend:0.0.0-alpha.13
imagePullPolicy: Always
ports:
- containerPort: 8000
env:
- name: REDIS_HOST
value: 'myapp-redis'
- name: REDIS_PASSWORD
value: 'test'
imagePullSecrets:
- name: regsecret
---
Python 后端 pod 描述
这是我从kubectl describe po myapp-backend-58... -n mw-dev得到的:
Name: myapp-backend-585d...
Namespace: mw-dev
Node: worker-2/ip...
Start Time: Sat, 03 Feb 2018 13:08:01 +0100
Labels: name=myapp-backend
pod-template-hash=myhash
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"mw-dev","name":"myapp-backend-58...","uid":"e13...
Status: Running
IP: 10.2.2.180
Controlled By: ReplicaSet/myapp-backend-58...
Containers:
myapp-backend:
Container ID: docker://78cfc218d...
Image: registry/myapp-backend:0.0.0-alpha.13
Image ID: docker-pullable://registry/mw-dev/myapp-backend@sha256:785a...
Port: 8000/TCP
State: registryg
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Sat, 03 Feb 2018 13:55:07 +0100
Finished: Sat, 03 Feb 2018 13:55:08 +0100
Ready: False
Restart Count: 14
Environment:
REDIS_HOST: myapp-redis
REDIS_PASSWORD: test
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7... (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-7cm7c:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7...
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50m default-scheduler Successfully assigned myapp-backend-58... to worker-2
Normal SuccessfulMountVolume 50m kubelet, worker-2 MountVolume.SetUp succeeded for volume "default-token-7..."
Warning BackOff 50m (x4 over 50m) kubelet, worker-2 Back-off restarting failed container
Normal Pulling 50m (x4 over 50m) kubelet, worker-2 pulling image "registry/mw-dev/myapp-backend:0.0.0-alpha.13"
Normal Pulled 50m (x4 over 50m) kubelet, worker-2 Successfully pulled image "registry/mw-dev/myapp-backend:0.0.0-alpha.13"
Normal Created 50m (x4 over 50m) kubelet, worker-2 Created container
Normal Started 50m (x4 over 50m) kubelet, worker-2 Started container
Warning FailedSync 52s (x229 over 50m) kubelet, worker-2 Error syncing pod
正在运行的 pod
kubectl get pods --all-namespaces:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cert-manager-cert-manager-59fff59c7b-vdnd7 2/2 Running 4 3d
kube-system digitalocean-cloud-controller-manager-6d6b675bfd-nxqq2 1/1 Running 0 3d
kube-system digitalocean-provisioner-d4c79dfb4-mhb5d 1/1 Running 0 3d
kube-system heapster-56bf7c7896-9rv4z 1/1 Running 0 3d
kube-system kube-apiserver-wp7b4 1/1 Running 5 10d
kube-system kube-controller-manager-586c9b745b-gkqk4 1/1 Running 2 10d
kube-system kube-controller-manager-586c9b745b-pdhw7 1/1 Running 1 10d
kube-system kube-dns-7d74988c8b-z9zs2 3/3 Running 0 10d
kube-system kube-flannel-5wlk6 2/2 Running 0 10d
kube-system kube-flannel-khsvq 2/2 Running 0 10d
kube-system kube-flannel-skt2m 2/2 Running 4 10d
kube-system kube-proxy-cwqv8 1/1 Running 2 10d
kube-system kube-proxy-mg8jx 1/1 Running 0 10d
kube-system kube-proxy-vmw8g 1/1 Running 0 10d
kube-system kube-scheduler-7686847675-5kkhn 1/1 Running 1 10d
kube-system kube-scheduler-7686847675-lkm98 1/1 Running 2 10d
kube-system kubernetes-dashboard-7658f8d76-svtzh 1/1 Running 0 3d
kube-system loadbalancer-nginx-ingress-controller-8649c7986b-jndzz 1/1 Running 3 3d
kube-system loadbalancer-nginx-ingress-default-backend-6fb9444c64-bpz4g 1/1 Running 0 3d
kube-system pod-checkpointer-kfcpp 1/1 Running 0 10d
kube-system pod-checkpointer-kfcpp-spc1aitu1i-master-1 1/1 Running 0 10d
kube-system tiller-deploy-fb8d7b69c-6xrpn 1/1 Running 2 3d
mw-dev myapp-backend-6c4b56d9b7-2mfbs 1/1 Running 0 21m
mw-dev myapp-frontend-7478fd456b-5ztvq 1/1 Running 0 1d
mw-dev myapp-redis-67d45d97d7-7wxtj 1/1 Running 0 1d
确保 Python 应用收到正确的环境变量值
Python 应用程序会打印出用于连接数据库的值。查看 pod 日志,我可以看到这些值与 Backend.yaml(REDIS_HOST=myapp-redis、REDIS_PASSWORD=test)中给出的值相同。
它在 docker 本地工作
如果我在笔记本电脑上使用 docker 在本地运行 redis 容器和 python 应用程序容器,它们可以正常连接。
集群信息
集群使用 nginx-ingress 控制器向互联网公开服务。我不确定这是否重要,因为我需要在内部将 Python pod 连接到 redis 服务。
集群由 1 个主节点、两个 Worker 节点和一个用于 nginx-ingress 控制器的 LoadBlancer 组成,所有这些都在 DigitalOcean 上运行。
现在怎么办?
此时我不知道如何进一步调试该问题。我已经在网上搜索了几个小时以找到解决方案,没有运气。任何建议将不胜感激!
【问题讨论】:
-
如果您使用服务ClusterIP从后端pod ping redis数据库是否有效?
-
@fiunchinho 似乎。 ping 返回:
10.3.0.137:6379 (10.3.0.137): 56 data bytes -
我的意思是 redis-cli ping 命令。你能把你是怎么做的贴在这个问题上吗?
-
如果我从另一个 pod 运行这个:
redis-cli -h 10.3.0.137 -p 6379 -a test ping我得到这个Could not connect to Redis at 10.3.0.137:6379: Connection refused -
运行
kubectl get pods --all-namespaces以查看 kube-dns 等内部 kubernetes pod 是否正在运行。
标签: redis kubernetes