nginx 与 minikube 和 metallb答案

【问题标题】：nginx with minikube and metallbnginx 与 minikube 和 metallb
【发布时间】：2020-12-09 03:03:37
【问题描述】：

您好，我正在尝试在 minikube 中使用我自己的容器启动我自己的部署。这是我的 yaml 文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wildboar-nginx-depl
  labels:
    app: services.nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: services.nginx
  template:
    metadata:
      labels:
        app: services.nginx
    spec:
      containers:
        - name: wildboar-nginx-pod
          image: services.nginx
          ports:
            - containerPort: 80
            - containerPort: 443
            - containerPort: 22
          imagePullPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
  name: wildboar-nginx-service
  annotations: 
    metallb.universe.tf/allow-shared-ip: wildboar-key
spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.1.101 
  selector:
    app: services.nginx
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
      nodePort: 30080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
      nodePort: 30443
    - name: ssh
      protocol: TCP
      port: 22
      targetPort: 22
      nodePort: 30022

这是我的 Dockerfile

FROM alpine:latest
RUN apk update && apk upgrade -U -a
RUN apk add nginx openssl openrc openssh supervisor
RUN mkdir /www/
RUN adduser -D -g 'www' www
RUN chown -R www:www /www
RUN chown -R www:www /var/lib/nginx
RUN openssl req -x509 -nodes -days 30 -newkey rsa:2048 -subj \
"/C=RU/ST=Moscow/L=Moscow/O=lchantel/CN=localhost" -keyout \
/etc/ssl/private/lchantel.key -out /etc/ssl/certs/lchantel.crt
COPY ./conf /etc/nginx/conf.d/default.conf
COPY ./nginx_conf.sh .
COPY ./supervisor.conf /etc/
RUN mkdir -p /run/nginx/
EXPOSE 80 443 22
RUN chmod 755 /nginx_conf.sh
CMD sh nginx_conf.sh

那是我的 nginx_conf.sh

#!bin/sh

cp /var/lib/nginx/html/index.html /www/
rc default
rc-service sshd start
ssh-keygen -A
rc-service sshd stop
/usr/bin/supervisord -c /etc/supervisord.conf

在我成功应用 yaml 文件后，我陷入了 CrashLoopBackOff 错误：

$ kubectl get pod
NAME                                   READY   STATUS             RESTARTS   AGE
wildboar-nginx-depl-57d64f58d8-cwcnn   0/1     CrashLoopBackOff   2          40s
wildboar-nginx-depl-57d64f58d8-swmq2   0/1     CrashLoopBackOff   2          40s

我尝试重新启动，但没有帮助。我试图描述 pod，但信息没有帮助：

$ kubectl describe pod wildboar-nginx-depl-57d64f58d8-cwcnn
Name:         wildboar-nginx-depl-57d64f58d8-cwcnn
Namespace:    default
Priority:     0
Node:         minikube/192.168.99.100
Start Time:   Sun, 06 Dec 2020 17:49:19 +0300
Labels:       app=services.nginx
              pod-template-hash=57d64f58d8
Annotations:  <none>
Status:       Running
IP:           172.17.0.7
IPs:
  IP:           172.17.0.7
Controlled By:  ReplicaSet/wildboar-nginx-depl-57d64f58d8
Containers:
  wildboar-nginx-pod:
    Container ID:   docker://6bd4ab3b08703293697d401e355d74d1ab09f938eb23b335c92ffbd2f8f26706
    Image:          services.nginx
    Image ID:       docker://sha256:a62f240db119e727935f072686797f5e129ca44cd1a5f950e5cf606c9c7510b8
    Ports:          80/TCP, 443/TCP, 22/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 06 Dec 2020 17:52:13 +0300
      Finished:     Sun, 06 Dec 2020 17:52:15 +0300
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 06 Dec 2020 17:50:51 +0300
      Finished:     Sun, 06 Dec 2020 17:50:53 +0300
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hr82j (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-hr82j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hr82j
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  3m9s                                    Successfully assigned default/wildboar-nginx-depl-57d64f58d8-cwcnn to minikube
  Normal   Pulled     98s (x5 over 3m9s)   kubelet, minikube  Container image "services.nginx" already present on machine
  Normal   Created    98s (x5 over 3m9s)   kubelet, minikube  Created container wildboar-nginx-pod
  Normal   Started    98s (x5 over 3m9s)   kubelet, minikube  Started container wildboar-nginx-pod
  Warning  BackOff    59s (x10 over 3m4s)  kubelet, minikube  Back-off restarting failed container

我没有想法该怎么办:(

【问题讨论】：

查看你的 pod 的kubectl logs；在没有 Kubernetes 的普通 Docker 上尝试 docker run 相同的容器；从镜像中删除不必要的部分（supervisord，sshd；仅运行 nginx，并作为前台进程），以便调试更简单。
好吧，我尝试使用 supervisord、sshd 和 nginx 本身运行它，它可以工作docker run -p 80:80 -p 443:433 -p 22:22 --name test proj.nginx:latest，但 pod 仍然会一直重启。我用kubectl logs 检查了 pod，它输出 pod 运行 nginx_config.sh 中的所有内容，除了主管......
添加到容器规范command: ['nginx', '-g', 'daemon off;']会更好吗？您是否会完全怀念容器中的 ssh 守护进程（在不太可能需要调试 shell 的情况下，您仍然可以使用 kubectl exec）？
我认为 "supervisor.conf" 的内容有些问题，因为 pod 正在进入完成状态。
@confusedgenius 你是对的。首先，我没有在 Dockerfile 中正确复制 supervisor.conf：它应该是 COPY ./supervisor.conf /etc/supervisord.conf，但我有 COPY ./supervisor.conf /etc/。 Docker 镜像和容器现在工作正常，但 pod 仍然不断重启。甚至kubectl logs <trouble_pod> 也没有向我展示任何东西。现在我正在尝试在没有主管和 ssh 密钥的情况下从头开始一步一步地编写它，但就像 David Maze 所说的那样，只有简单的 nginx

标签： docker kubernetes minikube metallb

【解决方案1】：

我解决了 nginx 的问题。首先，我重写了 supervisor.conf，现在是这样的：

[supervisord]
nodaemon=true
user = root

[program:nginx]
command=nginx -g 'daemon off;'
autostart=true
autorestart=true
startsecs=0
redirect_stderr=true

[program:ssh]
command=/usr/sbin/sshd -D
autostart=true
autorestart=true

第二，我遇到了 loadBalancer 的问题。我在文件中交换服务和部署配置，并为 service next stat spec.externalTrafficPolicy: Cluster (用于 IP 地址共享) 添加。

apiVersion: v1
kind: Service
metadata:
  name: wildboar-nginx-service
  labels:
    app: nginx
  annotations: 
    metallb.universe.tf/allow-shared-ip: minikube
spec:
  type: LoadBalancer
  loadBalancerIP: 192.168.99.105
  selector:
    app: nginx
  externalTrafficPolicy: Cluster
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
    - name: ssh
      protocol: TCP
      port: 22
      targetPort: 22

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wildboar-nginx-depl
  labels:
    app: nginx
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
     restartPolicy: Always
     containers:
       - name: wildboar-nginx-pod
         image: wildboar.nginx:latest
         ports:
           - containerPort: 80
             name: http
           - containerPort: 443
             name: https
           - containerPort: 22
             name: ssh
         imagePullPolicy: Never

第三个我用这样的脚本重建了 minikube 和所有配置

#!/bin/bash

kubectl ns default
kubectl delete deployment --all
kubectl delete service --all
kubectl ns metallb-system
kubectl delete configmap --all
kubectl ns default
docker rmi -f <your_custom_docker_image>
minikube stop
minikube delete 
minikube start --driver=virtualbox --disk-size='<your size>mb' --memory='<your_size>mb'
minikube addons enable metallb
eval $(minikube docker-env)
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# next line is only when you use mettallb for first time
#kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
docker build -t <your_custom_docker_images> .
kubectl apply -f <mettalb_yaml_config>.yaml
kubectl apply -f <your_config_with_deployment_and_service>.yaml

我还提到，yaml 文件对空格和制表符非常敏感，所以我安装了 yamlint 用于 yaml 文件的基本调试。我要感谢困惑的天才和大卫迷宫的帮助！

【讨论】：