Fluentd 无法使用 fluent-plugin-elasticsearch 连接到 AWS elasticsearch 服务

Question

我们正在使用 AWS Elasticsearch Service 和 fleuntd 来推送微服务的日志。我们已经使用基于 docker 的配置在 EC2 实例上安装了 fluentd。我们已经按照https://docs.fluentd.org/container-deployment/docker-compose 中提到的步骤进行了操作，并且直到上周它都运行良好。我们正在配置的 elasticsearch 服务和 ec2 实例的配置都没有变化。以下是我们得到的错误日志。

2021-08-05 20:35:57 +0000 [错误]：#0 意外错误 error_class=NoMethodError 错误="未定义方法 host_unreachable_exceptions' for #<Elasticsearch::Transport::Client:0x00007fe5231b5e08>" 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:41:in rescue in retry_operate' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/elasticsearch_index_template.rb:39:in retry_operate' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:487:in handle_last_seen_es_major_version' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:339:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in configure' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in add_match' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in block in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in each' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in configure' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:146:in configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in configure' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in run_configure' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:714:in block in run_worker' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:966:in main_process' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:706:in run_worker' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:364:in <top (required)>' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in require' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in ' 2021-08-05 20:35:57 +0000 [错误]：#0 /usr/bin/fluentd:23:in load' 2021-08-05 20:35:57 +0000 [error]: #0 /usr/bin/fluentd:23:in '

下面是使用的配置。

docker-compose.yml

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    ports:
      - "8880:8880"
      - "8880:8880/udp"

Dockerfile

FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent

fluentd.conf

<source>
  @type forward
  port 8880
  bind 0.0.0.0
</source>

<match **>
    host ES_HOT
    port 443
    scheme https
    logstash_format true
    logstash_prefix "fluentd-logs"
    logstash_dateformat "%Y%m%d"
    user "elastic"
    password xxxxxx
    flush_interval 10s
    fail_on_detecting_es_version_retry_exceed false
    verify_es_version_at_startup false
    max_retry_get_es_version 2
    default_elasticsearch_version 7.10
    request_timeout 60s
    reload_on_failure true
</match>

我们使用 AWS Elasticsearch Service 7.10、Fluentd v1.12.0-debian-1.0 和 fluent-plugin-elasticsearch 4.3.3 的配置，直到上周都工作得很好。 10 天前配置的 ec2 实例之一也能够连接和推送日志。但是当我们重新启动该实例时，它也停止工作并出现相同的错误。我们有提供所有基础设施的 Terraform 脚本。后来，我们也尝试了不同的版本组合。但没有任何结果。

提前致谢

Answer 1

我遇到了类似的问题。深入研究这个问题，我发现fluent-plugin-elasticsearch 插件不会对它的依赖项进行版本锁定，包括elasticsearch-transport 和elasticsearch-api，并且其中一个最近的更新导致了某种不兼容。

对我有用的修复是在最新的无问题版本中显式安装依赖项，以避免 fluent-plugin-elasticsearch 安装拉取最新的导致问题的版本。

对 Dockerfile 的建议修改

FROM fluent/fluentd:v1.13.3
USER root
RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "7.13.3"]
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"]
USER fluent