Kafka 连接 docker 映像 - 找不到任何实现连接器且名称与 ElasticsearchSinkConnector 匹配的类

Question

我已经使用 kafka-connect 图像confluentinc/cp-kafka-connect 有一段时间了。根据 Confluent 文档，这个 docker 镜像带有预安装的连接器插件，包括 Elastic。

我之前一直在使用 5.4.1-ccs 版本，它工作得很好，我可以添加 Elastic sink 连接器配置，它们工作得很好。但是我尝试将confluentinc/cp-kafka-connect 更新为最新的v6.0.1，现在我得到了错误。

ConnectException: Failed to find any class that implements Connector and which name matches ElasticsearchSinkConnector

我已经阅读了 Confluent 网站上的大量文档，但内容有点零星。我知道问题是插件未安装，因为它们要么从新的 docker 映像中删除，要么路径错误（不确定是哪一个）。

我该如何解决这个问题？ （注意：我也编写了自己的 java 插件，所以两者都需要工作）

这是我目前的docker-compose 文件（同样，这适用于版本5.4.1-ccs）

kafka-connect-node-1:
  image: confluentinc/cp-kafka-connect:5.4.1 #using old version because of breaking change
  hostname: kafka-connect-node-1
  ports:
    - '8083:8083'
  environment:
    CONNECT_BOOTSTRAP_SERVERS: [MY_SERVER]
    CONNECT_REST_PORT: 8083
    CONNECT_GROUP_ID: compose-connect-group
    CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
    CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
    CONNECT_STATUS_STORAGE_TOPIC: connect-status
    CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
    CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: 'http://kafka-schema-registry:8084'
    CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
    CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://kafka-schema-registry:8084'
    CONNECT_INTERNAL_KEY_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
    CONNECT_INTERNAL_VALUE_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
    CONNECT_REST_ADVERTISED_HOST_NAME: 'kafka-connect-node-1'
    CONNECT_LOG4J_ROOT_LOGLEVEL: 'INFO'
    CONNECT_LOG4J_LOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR'
    CONNECT_PLUGIN_PATH: '/usr/share/java,/etc/kafka-connect/jars'
    CONNECT_ZOOKEEPER_CONNECT: [MY_ZOOKEEPER]
  volumes:
    - /efs/connector:/etc/kafka-connect/jars/ # mounting my custom JAR
  depends_on:
    - kafka-schema-registry
    - kafka-rest-proxy

Answer 1

Since Confluent Platform 6.0 connectors are no longer bundled，需要单独安装。

您可以在cp-kafka-connect-base 的基础上使用build your own image，也可以在运行时通过覆盖图像命令thus 来安装连接器：

  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.0.0
    container_name: kafka-connect
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: _kafka-connect-group-01-configs
      CONNECT_OFFSET_STORAGE_TOPIC: _kafka-connect-group-01-offsets
      CONNECT_STATUS_STORAGE_TOPIC: _kafka-connect-group-01-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: '/usr/share/java,/connectors,/usr/share/confluent-hub-components/'
    depends_on:
      - broker
      - schema-registry
    volumes:
      - $PWD/data/connectors/:/connectors/
    command:
      - bash
      - -c
      - |
        echo "Installing Connector"
        confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:10.0.1
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run &
        #
        sleep infinity