k8s 上的 RabbitMQ 不断重启 - 找不到 rabbitmq-diagnostics答案

【问题标题】：RabbitMQ on k8s keeps restarting - not able to find rabbitmq-diagnosticsk8s 上的 RabbitMQ 不断重启 - 找不到 rabbitmq-diagnostics
【发布时间】：2022-01-27 07:59:51
【问题描述】：

尝试在本地 k8s 集群上运行 RabbitMQ，但它一直在重启，直到 CrashLoopBackOff 出现错误：

2021-12-28 18:11:50.098771+00:00 [erro] <0.846.0> Error on AMQP connection <0.846.0> (10.1.0.159:52048 -> 10.1.0.156:5672 - rabbitConnectionFactory#5dcbb60:0, vhost: '/', user: 'guest', state: running), channel 0:
2021-12-28 18:11:50.098771+00:00 [erro] <0.846.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2021-12-28 18:11:50.098771+00:00 [erro] <0.1089.0> Error on AMQP connection <0.1089.0> (10.1.0.158:54532 -> 10.1.0.156:5672 - rabbitConnectionFactory#102cec62:0, vhost: '/', user: 'guest', state: running), channel 0:
2021-12-28 18:11:50.098771+00:00 [erro] <0.1089.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"
2021-12-28 18:11:50.098888+00:00 [erro] <0.820.0> Error on AMQP connection <0.820.0> (10.1.0.155:51038 -> 10.1.0.156:5672 - rabbitConnectionFactory#67b7c170:10, vhost: '/', user: 'guest', state: running), channel 0:
2021-12-28 18:11:50.098888+00:00 [erro] <0.820.0>  operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"

服务：

apiVersion: v1
kind: Service
metadata:
  annotations:
    appName: {{ include "common.fullname" . }}
    componentName: rabbitmq
  labels:
    io.kompose.service: rabbitmq
  name: rabbitmq
spec:
  ports:
    - name: "{{ .Values.service.ports.rabbitmq.port1 }}"
      port: {{ .Values.service.ports.rabbitmq.port1 }}
      targetPort: {{ .Values.service.ports.rabbitmq.port1 }}
    - name: "{{ .Values.service.ports.rabbitmq.port2 }}"
      port: {{ .Values.service.ports.rabbitmq.port2 }}
      targetPort: {{ .Values.service.ports.rabbitmq.port2 }}
  type: LoadBalancer
  selector:
    io.kompose.service: rabbitmq
status:
  loadBalancer: {}

部署：

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    appName: {{ include "common.fullname" . }}
    componentName: rabbitmq
  labels:
    io.kompose.service: rabbitmq
    {{- include "common.labels" . | nindent 4 }}
  name: rabbitmq
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      io.kompose.service: rabbitmq
  strategy: {}
  template:
    metadata:
      annotations:
        appName: {{ include "common.fullname" . }}
        componentName: rabbitmq
      labels:
        io.kompose.service: rabbitmq
    spec:
      containers:
        - env:
            - name: DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: direct-secrets
                  key: rabbitmq-password
          envFrom:
            - configMapRef:
                name: rabbitmq-configmap
          image: rabbitmq:3-management
          livenessProbe:
            exec:
              command:
                - rabbitmq-diagnostics -q ping
            failureThreshold: 3
            periodSeconds: 30
            timeoutSeconds: 30
          name: rabbitmq
          ports:
            - containerPort: {{ .Values.app.rabbitmq.port1 }}
            - containerPort: {{ .Values.app.rabbitmq.port2 }}
          resources: {}
      restartPolicy: Always
status: {}

配置图：

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-configmap
data:
  DEFAULT_USER: "{{ .Values.app.rabbitmq.user }}"

价值观：

app:

  rabbitmq:
    host: rabbitmq
    port1: 5672
    port2: 15672
    password: guest
    user: guest

service:
  ports:
    rabbitmq:
      port1: 5672
      targetPort1: 5672
      port2: 15672
      targetPort2: 15672

对此有什么想法吗？

PS。

来自 pod describe 的输出：

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  17m                 default-scheduler  Successfully assigned default/rabbitmq-df54875b8-6m4dq to docker-desktop
  Normal   Killing    12m (x3 over 15m)   kubelet            Container rabbitmq failed liveness probe, will be restarted
  Normal   Pulled     11m (x4 over 16m)   kubelet            Container image "rabbitmq:3-management" already present on machine
  Normal   Created    11m (x4 over 16m)   kubelet            Created container rabbitmq
  Normal   Started    11m (x4 over 16m)   kubelet            Started container rabbitmq
  Warning  Unhealthy  95s (x24 over 16m)  kubelet            Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "rabbitmq-diagnostics -q ping": executable file not found in $PATH: unknown

由于某种原因无法找到以前图像中出现的rabbitmq-diagnostics。

【问题讨论】：

pod 正在重启吗？其中一个吊舱上的describe 是否显示任何其他线索？
提供 describe 命令的输出
@RakeshGupta OP 更新
@jordanm OP 已更新 - 在图片中找不到 rabbitmq-diagnostics
显然 rabbitmq:3-management 映像没有 liveness probe 试图调用的二进制 rabbitmq-diagnostics。请改用标准图片。

标签： kubernetes rabbitmq

【解决方案1】：

发现图像包含rabbitmq-diagnostics，但是当您将命令指定为这样的单个字符串时，它会在路径中寻找一个二进制文件，该文件实际上是完整的字符串，包括空格，rabbitmq-diagnostics -q ping，所以它应该是像这样的东西：

          command:
            - rabbitmq-diagnostics
            - -q
            - ping

或：

          command: ['rabbitmq-diagnostics', '-q', 'ping']

【讨论】：