Elasticsearch Pod 在没有日志的 Init 状态后失败

Question

我正在尝试让 Elasticsearch StatefulSet 在 AKS 上工作，但 pod 失败并在我能够看到任何日志之前被终止。有没有办法在 Pod 终止后查看日志？

这是我使用kubectl apply -f es-statefulset.yaml 运行的示例 YAML 文件：

# RBAC authn and authz
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: kube-system
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: elasticsearch-logging
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: elasticsearch-logging
  apiGroup: ""
---
# Elasticsearch deployment itself
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v6.4.1
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  serviceName: elasticsearch-logging
  replicas: 2
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
      version: v6.4.1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v6.4.1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: elasticsearch-logging
      containers:
      - image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1
        name: elasticsearch-logging
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: "1000m"
            memory: "2048Mi"
          requests:
            cpu: "100m"
            memory: "1024Mi"
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-logging
          mountPath: /data
        env:
        - name: "NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: "bootstrap.memory_lock"
          value: "true"
        - name: "ES_JAVA_OPTS"
          value: "-Xms1024m -Xmx2048m"
        - name: "discovery.zen.ping.unicast.hosts"
          value: "elasticsearch-logging"
      # A) This volume mount (emptyDir) can be set whenever not working with a
      # cloud provider. There will be no persistence. If you want to avoid
      # data wipeout when the pod is recreated make sure to have a
      # "volumeClaimTemplates" in the bottom.
      # volumes:
      # - name: elasticsearch-logging
      #   emptyDir: {}
      #
      # Elasticsearch requires vm.max_map_count to be at least 262144.
      # If your OS already sets up this number to a higher value, feel free
      # to remove this init container.
      initContainers:
      - image: alpine:3.6
        command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
        name: elasticsearch-logging-init
        securityContext:
          privileged: true
  # B) This will request storage on Azure (configure other clouds if necessary)
  volumeClaimTemplates:
    - metadata:
        name: elasticsearch-logging
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: default
        resources:
          requests:
            storage: 64Gi

当我“关注”时，创建的 pod 看起来像这样：

我尝试通过执行logs -n kube-system elasticsearch-logging-0 -p 并注释来从终止的实例中获取日志。

我正在尝试在 this sample from the official (unmaintained) k8s repo 之上构建。一开始是可行的，但是在我尝试更新部署后，我发现它完全失败了，我无法恢复它。我正在使用 Azure AKS 的试用版

感谢任何建议

编辑 1：

kubectl describe statefulset elasticsearch-logging -n kube-system 的结果如下（具有几乎相同的 Init-Terminated pod 流）：

Name:               elasticsearch-logging
Namespace:          kube-system
CreationTimestamp:  Mon, 24 Sep 2018 10:09:07 -0600
Selector:           k8s-app=elasticsearch-logging,version=v6.4.1
Labels:             addonmanager.kubernetes.io/mode=Reconcile
                    k8s-app=elasticsearch-logging
                    kubernetes.io/cluster-service=true
                    version=v6.4.1
Annotations:        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"elasticsea...
Replicas:           0 desired | 1 total
Update Strategy:    RollingUpdate
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=elasticsearch-logging
                    kubernetes.io/cluster-service=true
                    version=v6.4.1
  Service Account:  elasticsearch-logging
  Init Containers:
   elasticsearch-logging-init:
    Image:      alpine:3.6
    Port:       <none>
    Host Port:  <none>
    Command:
      /sbin/sysctl
      -w
      vm.max_map_count=262144
    Environment:  <none>
    Mounts:       <none>
  Containers:
   elasticsearch-logging:
    Image:       docker.elastic.co/elasticsearch/elasticsearch:6.4.1
    Ports:       9200/TCP, 9300/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     1
      memory:  2Gi
    Requests:
      cpu:     100m
      memory:  1Gi
    Environment:
      NAMESPACE:                          (v1:metadata.namespace)
      bootstrap.memory_lock:             true
      ES_JAVA_OPTS:                      -Xms1024m -Xmx2048m
      discovery.zen.ping.unicast.hosts:  elasticsearch-logging
    Mounts:
      /data from elasticsearch-logging (rw)
  Volumes:  <none>
Volume Claims:
  Name:          elasticsearch-logging
  StorageClass:  default
  Labels:        <none>
  Annotations:   <none>
  Capacity:      64Gi
  Access Modes:  [ReadWriteMany]
Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  53s   statefulset-controller  create Pod elasticsearch-logging-0 in StatefulSet elasticsearch-logging successful
  Normal  SuccessfulDelete  1s    statefulset-controller  delete Pod elasticsearch-logging-0 in StatefulSet elasticsearch-logging successful

流程保持不变：

Answer 1

您假设 pod 因 ES 相关错误而终止。
我不太确定 ES 是否开始运行，这应该可以解释缺少日志的原因。

拥有多个同名的 pod 非常可疑，尤其是在 StatefulSet 中，所以那里出了点问题。
我会先尝试kubectl describe statefulset elasticsearch-logging -n kube-system，这应该可以解释发生了什么——可能是在安装卷之前运行ES的一些问题。

我也很确定你想将 ReadWriteOnce 更改为 ReadWriteMany。

希望这会有所帮助！

Answer 2

是的。有办法。您可以 ssh 进入运行您的 pod 的机器，假设您使用的是 Docker，您可以运行：

docker ps -a # Shows all the Exited containers (some of those, part of your pod)

然后：

docker logs <container-id-of-your-exited-elasticsearch-container>

如果您使用的是CRIO 或Containerd，这也可以使用，类似于

crictl logs <container-id>