如何从 HA 集群和 etcd 集群中删除主节点

Question

我是 k8s 新手，发现了一个我无法解决的问题。

我正在构建一个主节点的 HA 集群。我正在运行一些测试（删除一个节点并再次添加该节点）。通过这个过程我注意到etcd集群没有更新集群列表。

以下问题示例：

$ kubectl get pods -A
NAMESPACE                NAME                                                 READY   STATUS    RESTARTS   AGE
cri-o-metrics-exporter   cri-o-metrics-exporter-77c9cf9746-qlp4d              0/1     Pending   0          16h
haproxy-controller       haproxy-ingress-769d858699-b8r8q                     0/1     Pending   0          16h
haproxy-controller       ingress-default-backend-5fd4986454-kvbw8             0/1     Pending   0          16h
kube-system              calico-kube-controllers-574d679d8c-tkcjj             1/1     Running   3          16h
kube-system              calico-node-95t6l                                    1/1     Running   2          16h
kube-system              calico-node-m5txs                                    1/1     Running   2          16h
kube-system              coredns-7588b55795-gkfjq                             1/1     Running   2          16h
kube-system              coredns-7588b55795-lxpmj                             1/1     Running   2          16h
kube-system              etcd-masterNode1                                     1/1     Running   2          16h
kube-system              etcd-masterNode2                                     1/1     Running   2          16h
kube-system              kube-apiserver-masterNode1                           1/1     Running   3          16h
kube-system              kube-apiserver-masterNode2                           1/1     Running   3          16h
kube-system              kube-controller-manager-masterNode1                  1/1     Running   4          16h
kube-system              kube-controller-manager-masterNode2                  1/1     Running   4          16h
kube-system              kube-proxy-5q6xs                                     1/1     Running   2          16h
kube-system              kube-proxy-k8p6h                                     1/1     Running   2          16h
kube-system              kube-scheduler-masterNode1                           1/1     Running   3          16h
kube-system              kube-scheduler-masterNode2                           1/1     Running   6          16h
kube-system              metrics-server-575bd7f776-jtfsh                      0/1     Pending   0          16h
kubernetes-dashboard     dashboard-metrics-scraper-6f78bc588b-khjjr           1/1     Running   2          16h
kubernetes-dashboard     kubernetes-dashboard-978555c5b-9jsxb                 1/1     Running   2          16h
$ kubectl exec etcd-masterNode2 -n kube-system -it -- sh
sh-5.0# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list -w table
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |            NAME            |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
| 4c209e5bc1ca9593 | started |         masterNode1        |     https://IP1:2380      |     https://IP1:2379      |      false |
| 676d4bfab319fa22 | started |         masterNode2        |     https://IP2:2380      |     https://IP2:2379      |      false |
| a9af4b00e33f87d4 | started |         masterNode3        |     https://IP3:2380      |     https://IP3:2379      |      false |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
sh-5.0# exit
$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
masterNode1                  Ready    master   16h   v1.19.0
masterNode2                  Ready    master   16h   v1.19.0

我假设我从集群中正确删除了节点。我遵循的程序：

1. kubectl drain --ignore-daemonsets --delete-local-data
2. kubectl 删除
3. 节点 kubeadm 重置
4. rm -f /etc/cni/net.d/* #删除CNI配置
5. rm -rf /var/lib/kubelet # 删除 /var/lib/kubeler 目录
6. rm -rf /var/lib/etcd # 删除 /var/lib/etcd
7. iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X && iptables -t filter -F && iptables -t filter -X # 删除iptables
8. ipvsadm --clear
9. rm -rf /etc/kubernetes # 删除/etc/kubernetes（以防字符变化）

我正在运行版本为 1.19.0 和 etcd etcd:3.4.9-1 的 kubernetes。

集群在裸机节点上运行。

这是一个错误还是我没有从 etcd 集群中正确删除节点？

Answer 1

感谢Mariusz K. 我找到了问题的答案。万一其他人可能有同样的问题，我就是这样解决的。

首先在集群 (HA) 中查询 etcd 成员（代码示例）：

$ kubectl exec etcd-< nodeNameMasterNode > -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
1863b58e85c8a808, started, nodeNameMaster1, https://IP1:2380, https://IP1:2379, false
676d4bfab319fa22, started, nodeNameMaster2, https://IP2:2380, https://IP2:2379, false
b0c50c50d563ed51, started, nodeNameMaster3, https://IP3:2380, https://IP3:2379, false

然后，一旦您有了节点列表，您就可以删除任何您想要的成员。代码示例：

kubectl exec etcd-nodeNameMaster1 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove b0c50c50d563ed51
Member b0c50c50d563ed51 removed from cluster d1e1de99e3d19634

我希望能够从 etcd 集群中删除一个成员，而无需连接到 pod 并运行辅助命令。这样我通过exec执行命令到pod。