【发布时间】：2020-06-13 16:32:17
【问题描述】：
编辑：我根据efrat-levitan's 的建议添加了新信息。当我按照原始评论的建议将 Elasticsearch 版本升级到 7.6.0 时，列出的日志输出略有不同。为了帮助调试，我也没有立即启动 Elasticsearch。可以在日志中的ECONNREFUSED 消息中看到这样做的效果。我在下面的摘要中调用了日志文件更改。其余大部分消息文本（即不记录 sn-ps）与以前保持一致。
我一直在努力获得 fluentd 的 ARM 版本（用于 Raspberry Pi 3 和 4），并在 docker 中运行 fluent-plugin-elasticsearch 插件。我找不到合适的 docker 镜像，所以我自己构建了一个（如果有人知道我在哪里可以找到，我会很感激）。我从 fluentd-docker-image 存储库（不包括 Elasticsearch 插件）开始，并根据我认为必要的情况使用 fluentd-kubernetes-daemonset 存储库（包括 Elasticsearch 插件）对其进行了修改。好消息是它在 Raspberry Pi 上启动得很好。坏消息是它似乎甚至没有尝试连接到 ElasticSearch（在 Raspberry Pi 网络外部。日志文件如下所示：
2020-03-02 18:13:15 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.12'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.4'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-03-02 18:13:16 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.1'
2020-03-02 18:13:16 +0000 [info]: gem 'fluentd' version '1.9.2'
2020-03-02 18:13:16 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:16 +0000 [info]: using configuration file: <ROOT>
<filter **>
@type stdout
</filter>
<source>
@type forward
@label @ES
</source>
<label @ES>
<match out.elasticsearch.**>
@type elasticsearch
@log_level "info"
include_tag_key true
host "10.0.0.223"
port 9200
path ""
scheme http
index_name "logstash"
include_timestamp true
log_es_400_reason false
logstash_prefix "logstash"
logstash_dateformat "%Y.%m.%d"
logstash_format true
ssl_verify true
ssl_version TLSv1_2
user
password xxxxxx
reload_connections false
reconnect_on_error true
reload_on_failure true
request_timeout 5s
sniffer_class_name "Fluent::Plugin::ElasticsearchSimpleSniffer"
type_name "doc"
template_name
template_file
template_overwrite false
time_key "@timestamp"
<buffer>
flush_thread_count 8
flush_interval 5s
chunk_limit_size 2M
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
</match>
</label>
<label @ERROR>
<match **>
@type stdout
</match>
</label>
</ROOT>
2020-03-02 18:13:16 +0000 [info]: starting fluentd-1.9.2 pid=7 ruby="2.6.5"
2020-03-02 18:13:16 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "-r", "/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:23 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2020-03-02 18:13:27 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:27 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2020-03-02 18:13:35 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:35 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
2020-03-02 18:13:51 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-03-02 18:13:51 +0000 [info]: adding match in @ERROR pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding filter pattern="**" type="stdout"
2020-03-02 18:13:51 +0000 [info]: adding source type="forward"
2020-03-02 18:13:51 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-03-02 18:13:51 +0000 [info]: #0 starting fluentd worker pid=22 ppid=7 worker=0
2020-03-02 18:13:51 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2020-03-02 18:13:51 +0000 [info]: #0 fluentd worker is now running worker=0
2020-03-02 18:13:51.581170450 +0000 fluent.info: {"pid":22,"ppid":7,"worker":0,"message":"starting fluentd worker pid=22 ppid=7 worker=0"}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.585716902 +0000 fluent.info: {"port":24224,"bind":"0.0.0.0","message":"listening port port=24224 bind=\"0.0.0.0\""}
2020-03-02 18:13:51 +0000 [warn]: #0 no patterns matched tag="fluent.info"
2020-03-02 18:13:51.593737828 +0000 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
为了进一步缩小日志范围，fluentd 似乎在配置和连接方面都了解 Elasticsearch：
2020-03-02 18:13:20 +0000 [info]: adding match in @ES pattern="out.elasticsearch.**" type="elasticsearch"
2020-03-02 18:13:23 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. Connection refused - connect(2) for 10.0.0.223:9200 (Errno::ECONNREFUSED)
“ECONNREFUSED”错误是因为我停止了 Elasticsearch 以帮助进行故障排除。所以 Fluentd 正在尝试连接。如下sn-p所示，启动Elasticsearch后，确实连接并继续处理：
2020-03-02 18:13:51 +0000 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
问题在于，Fluentd 似乎并没有真正完成 Elasticsearch 的“签名”过程。我希望在成功时看到类似的内容，或某种错误消息。
2020-02-28 21:56:26 +0000 [info]: #0 [out_es] Connection opened to Elasticsearch cluster => {:host=>"10.0.0.223", :port=>9200, :scheme=>"http", :path=>""}
我还希望在 Elasticsearch 中看到一些证据，证明它确实完成了“登录”过程。例如，我在 Elasticsearch 中没有看到“logstash”索引，也没有在 Elasticsearch 日志中看到任何证据表明除了 Kibana 之外的任何客户端都已连接。
上面记录了配置，对我来说似乎是正确的。日志中记录的命令行对我来说也很好。
/usr/local/bin/ruby -Eascii-8bit:ascii-8bit /usr/local/bundle/bin/fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -r /usr/local/bundle/gems/fluent-plugin-elasticsearch-4.0.4/lib/fluent/plugin/elasticsearch_simple_sniffer.rb --under-supervisor
登录 pod 并连接到 Elasticsearch 也可以：
$ ks exec -it fluentd-h2qzn sh
$ curl http://10.0.0.223:9200
{
"name" : "Richs-MacBook.local",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "OkZ2-Lj2RjW-pVyVl0C7og",
"version" : {
"number" : "7.6.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "7f634e9f44834fbc12724506cc1da681b0c3b1e3",
"build_date" : "2020-02-06T00:09:00.449973Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
有人可以说明缺少与 Elasticsearch 主机的连接，或者提供一些我可以执行的其他故障排除步骤吗？
谢谢， 丰富
【问题讨论】：
-
Detected ES 6.x: ES 7.x will only accept _doc in type_name看起来您正在使用 2 个不兼容的 ES 版本 -
确实，这可能是问题的一部分，确实是版本不匹配。我安装了 Elasticsearch 7.6.0 和相应的 Kibana 版本 7.6.0，我可以看到 fluentd 正在尝试连接。为了帮助解决问题，我停止了 Elasticsearch 以查看日志的样子，我编辑了上面的帖子以包含更改。
-
我用不同的配置重启了 fluentd pod 几次。我终于看到它连接了，但是 elasticsearch 中的日志显示这发生在最后一次 pod 重启后的 2 小时内。嗯……
标签： docker elasticsearch kubernetes raspberry-pi fluentd