使用 kubernetes 部署 zookeeper 服务时出错

Question

使用 kubernetes 在我们的裸机集群中部署 zookeeper 服务时出现以下错误。 kube-dns 服务也在运行，但仍然无法识别zookeeper-1 服务。

2018-03-13 12:11:32,217 [myid:3] - WARN  [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer$QuorumServer@155] - Failed to resolve address: zookeeper-1
java.net.UnknownHostException: zookeeper-1: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:922)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1316)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1269)
        at java.net.InetAddress.getAllByName(InetAddress.java:1185)
        at java.net.InetAddress.getAllByName(InetAddress.java:1119)
        at java.net.InetAddress.getByName(InetAddress.java:1069)
        at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:148)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:407)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:426)
        at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:843)
        at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:822)

我们如何解决这个问题？

Answer 1

在 StatefulSet 中，在创建相关 Pod 之前，主机域名不存在。在这种情况下，这可能意味着 Pod zookeeper-1 还不存在。

默认情况下，Stateful set 中的 Pod 只有在前一个 Pod 启动并且 ReadinessProbe 成功时才会启动。这可能表明zookeeper-0 尚未准备好，因为它无法联系zookeeper-1。可以通过将Pod Management Policy 设置为Parallel 来更改启动行为。

在集群节点相互连接之前，可能有必要让 Zookeeper 集群处于未就绪状态（即没有获得任何流量）。同时，可能要求所有节点在实际准备好之前可以相互连接。我们解决了这个问题，为 zookeeper 引入了第二个服务，我们称之为zookeeper-discovery。这个有一个注释service.alpha.kubernetes.io/tolerate-unready-endpoints 设置为true，这意味着它在这种情况下忽略了准备情况。所有依赖服务仍使用默认服务zookeeper。在 statefulset 中使用zookeeper-discovery 服务很重要：

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
  name: zookeeper-discovery
  labels:
    app: zookeeper
spec:
  clusterIP: None
  selector:
    app: zookeeper
  ports:
  - port: 2181
    name: client
  - port: 2888
    name: peer
  - port: 3888
    name: leader-election
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper
  labels:
    app: zookeeper
spec:
  selector:
    app: zookeeper
  ports:
  - port: 2181
    name: client
---
apiVersion: apps/v1beta1
kind: StatefulSet

metadata:
  name: zookeeper
  labels:
    app: zookeeper

spec:
  serviceName: zookeeper-disovery
  replicas: 3

  template:
    metadata:
      name: zookeeper
      labels:
        app: zookeeper
    spec:
      containers:
<-- snip -->