Gitlab CI 中的 Kafka 和 Zookeeper答案

【问题标题】：Kafka & Zookeeper in Gitlab CIGitlab CI 中的 Kafka 和 Zookeeper
【发布时间】：2022-01-19 16:37:11
【问题描述】：

如果我的应用程序运行正常且没有任何问题，我正在尝试运行一个简单的测试。我的问题是，faust 在初始化时需要连接到 kafka - 所以我试图将 kafka 与 zookeeper 作为服务一起运行，但我无法正确连接它们。

错误：

2021-12-16T13:53:51.385341793Z [2021-12-16 13:53:51,385] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2021-12-16T13:53:51.391012666Z [2021-12-16 13:53:51,390] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.395158219Z [2021-12-16 13:53:51,395] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2021-12-16T13:53:51.399485772Z [2021-12-16 13:53:51,397] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:51.399499707Z java.net.UnknownHostException: zookeeper: Name or service not known
2021-12-16T13:53:51.399503169Z  at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2021-12-16T13:53:51.399506400Z  at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
2021-12-16T13:53:51.399509510Z  at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
2021-12-16T13:53:51.399512353Z  at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
2021-12-16T13:53:51.399531020Z  at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
2021-12-16T13:53:51.399534098Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
2021-12-16T13:53:51.399537044Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
2021-12-16T13:53:51.399540881Z  at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2021-12-16T13:53:51.399544771Z  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2021-12-16T13:53:51.399548877Z  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2021-12-16T13:53:51.399553025Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1207)
2021-12-16T13:53:51.406655054Z [2021-12-16 13:53:51,406] WARN Session 0x0 for sever zookeeper:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.406696302Z java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
2021-12-16T13:53:51.406703099Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
2021-12-16T13:53:51.406707676Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
2021-12-16T13:53:51.406711700Z  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
2021-12-16T13:53:51.406715631Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
2021-12-16T13:53:52.508636206Z [2021-12-16 13:53:52,508] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:52.508665462Z java.net.UnknownHostException: zookeeper

.gitlab-ci.yml:

.zoo_service: &zoo_service
  name: zookeeper:latest
  alias: zookeeper

.kafka_service: &kafka_service
  name: bitnami/kafka:latest
  alias: kafka


faust:
  variables:
    ALLOW_ANONYMOUS_LOGIN: "yes"
    KAFKA_BROKER_ID: 1
    KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092"
    KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://127.0.0.1:9092"
    KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ALLOW_PLAINTEXT_LISTENER: "yes"
  stage: test
  <<: *python_image
  services:
    - *zoo_service
    - *kafka_service
  before_script:
    - *setup_venv_script
  script:
    - faust -A runner worker -l info & sleep 15; kill -HUP $!
  <<: *load_env
  except:
    - schedules

我希望我以正确的方式做这件事 - 遗憾的是，我可以阅读的关于这个问题的资源并不多。我知道问题出在 kafka 和 zookeeper 之间，但我不知道如何解决它（认为这是正确的方法）。甚至 2 个服务可以在 CI 中相互通信吗？

谢谢！

【问题讨论】：

为什么faust 服务有variables 用于kafka_service？
@OneCricketeer 我在 faust 工作中启动了服务块内部的 kafka_service，我认为它得到了工作变量 - 所以这并不重要，我认为没有问题。
我不熟悉 gitlab-ci YAML 的语法，但似乎类似于 docker-compose，我猜......所以，1）确保所有服务都在同一个容器网络中启动2) 确保 alias 标记是在该网络中实际设置容器的主机名
还有，和当前的错误无关，不过以后会出问题——KAFKA_CFG_ADVERTISED_LISTENERS应该是PLAINTEXT://kafka:9092
@OneCricketeer 我认为FF_NETWORK_PER_BUILD 有效。将其添加到您的答案中，以便我接受。 :) 非常感谢！

标签： apache-kafka gitlab-ci apache-zookeeper

【解决方案1】：

浏览GitLab CI docs关于连接不同服务的信息，它提到了一个允许跨服务通信的功能标志，所以试试

faust:
  variables:
    FF_NETWORK_PER_BUILD: 1
    ...
  services:
    ...

另外，对于Kafka通信，它需要通告它的别名而不是本地主机，所以改变

KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"

【讨论】：