Kubernetes 不断杀死并重新创建最后一个 pod

Question

最后一个（第三个）容器不断被 kubernetes 删除和重新创建。它从运行状态变为终止状态。 Kubernetes UI 显示状态为：'Terminated: ExitCode:${state.terminated.exitCode}'

我的部署 YAML：

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: openapi
spec:
  scaleTargetRef:
    kind: Deployment
    name: openapi
  minReplicas: 3
  maxReplicas: 10
targetCPUUtilizationPercentage: 75

---
kind: Service
apiVersion: v1
metadata:
  name: openapi
spec:
  selector:
    app: openapi
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443
 type: LoadBalancer

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: openapi
spec:
  template:
    metadata:
      labels:
        app: openapi
    spec:
      containers:
      - name: openapi
        image: us.gcr.io/PROJECT_ID/openapi:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080

kubectl get events -n namespace的输出部分：

Pod                       Normal    Created                 kubelet                     Created container
Pod                       Normal    Started                 kubelet                     Started container
Pod                       Normal    Killing                 kubelet                     Killing container with id docker://openapi:Need to kill Pod
ReplicaSet                Normal    SuccessfulCreate        replicaset-controller       (combined from similar events): Created pod: openapi-7db5f8d479-p7mcl
ReplicaSet                Normal    SuccessfulDelete        replicaset-controller       (combined from similar events): Deleted pod: openapi-7db5f8d479-pgmxf
HorizontalPodAutoscaler   Normal    SuccessfulRescale       horizontal-pod-autoscaler    New size: 2; reason: Current number of replicas above Spec.MaxReplicas
HorizontalPodAutoscaler   Normal    SuccessfulRescale       horizontal-pod-autoscaler    New size: 3; reason: Current number of replicas below Spec.MinReplicas
Deployment                Normal    ScalingReplicaSet       deployment-controller        Scaled up replica set openapi-7db5f8d479 to 3
Deployment                Normal    ScalingReplicaSet       deployment-controller        Scaled down replica set openapi-7db5f8d479 to 2

kubectl describe pod -n default openapi-7db5f8d479-2d2nm 用于生成并被杀死的 pod：

每次 pod 被 Kubernetes 杀死后，都会生成具有不同唯一 ID 的不同 pod。

Name:                      openapi-7db5f8d479-2d2nm
Namespace:                 default
Node:                      gke-testproject-default-pool-28ce3836-t4hp/10.150.0.2
Start Time:                Thu, 23 Nov 2017 11:50:17 +0000
Labels:                    app=openapi
                           pod-template-hash=3861948035
Annotations:               kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"openapi-7db5f8d479","uid":"b7b3e48f-ceb2-11e7-afe7-42010a960003"...
                           kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container openapi
Status:                    Terminating (expires Thu, 23 Nov 2017 11:51:04 +0000)
Termination Grace Period:  30s
IP:
Created By:                ReplicaSet/openapi-7db5f8d479
Controlled By:             ReplicaSet/openapi-7db5f8d479
Containers:
  openapi:
    Container ID:   docker://93d2f1372a7ad004aaeb34b0bc9ee375b6ed48609f505b52495067dd0dcbb233
    Image:          us.gcr.io/testproject-175705/openapi:latest
    Image ID:       docker-pullable://us.gcr.io/testproject-175705/openapi@sha256:54b833548cbed32db36ba4808b33c87c15c4ecde673839c3922577f30b
    Port:           8080/TCP
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Thu, 23 Nov 2017 11:50:18 +0000
      Finished:     Thu, 23 Nov 2017 11:50:35 +0000
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-61k6c (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-61k6c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-61k6c
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                                                Message
  ----    ------                 ----  ----                                                -------
  Normal  Scheduled              21s   default-scheduler                                   Successfully assigned openapi-7db5f8d479-2d2nm to gke-testproject-default-pool-28ce3836-t4hp
  Normal  SuccessfulMountVolume  21s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  MountVolume.SetUp succeeded for volume "default-token-61k6c"
  Normal  Pulling                21s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  pulling image "us.gcr.io/testproject-175705/openapi:latest"
  Normal  Pulled                 20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Successfully pulled image "us.gcr.io/testproject-175705/openapi:latest"
  Normal  Created                20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Created container
  Normal  Started                20s   kubelet, gke-testproject-default-pool-28ce3836-t4hp  Started container
  Normal  Killing                3s    kubelet, gke-testproject-default-pool-28ce3836-t4hp  Killing container with id docker://openapi:Need to kill Pod

Answer 1

使用以下命令检查 pod 日志： kubectl get events -w -n namespace 和 kubectl describe pod -n namespace pod_name