fluent-plugin-elasticsearch: "Could not push log to Elasticsearch" error with "error"=>{"type"=>"mapper_parsing_exception"}答案

【问题标题】：fluent-plugin-elasticsearch: "Could not push log to Elasticsearch" error with "error"=>{"type"=>"mapper_parsing_exception"}fluent-plugin-elasticsearch: "Could not push log to Elasticsearch" error with "error"=>{"type"=>"mapper_parsing_exception"}
【发布时间】：2017-09-04 14:49:55
【问题描述】：

当我使用 fluent-plugin-elasticsearch 将 Fluentd 收集的数据注入 Elasticsearch 时，一些数据导致以下错误：

2017-04-09 23:47:37 +0900 [error]: Could not push log to Elasticsearch: {"took"=>3, "errors"=>true, "items"=>[{"index"=>{"_index"=>"logstash-201704", "_type"=>"ruby", "_id"=>"AVtTLz_cUzkwT9CQCxrH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [message]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:27"}}}}, .....]}

elasticsearch 似乎因错误 failed to parse [message] 和 Can't get text on a START_OBJECT at 1:27 而禁止了数据。但我看不到哪些数据发送到 Elasticsearch 以及出了什么问题。

有什么想法吗？

【问题讨论】：

标签： elasticsearch fluent

【解决方案1】：

fluent-plugin-elasticsearch 使用 _bulk API 发送数据。我将请求转储代码放在/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-5.0.4/lib/elasticsearch/api/actions/bulk.rb 上，如下所示：

  def bulk(arguments={})
    ...
      payload = body
    end
    $log.info([method, path, params, payload].inspect)  # <=== here ($log is global logger of fluentd)
    perform_request(method, path, params, payload).body

我发现发送到 Elasticsearch 的请求如下：

POST /_bulk
{"index":{"_index":"logstash-201704","_type":"ruby"}}
{"level":"INFO","message":{"status":200,"time":{"total":46.26,"db":33.88,"view":12.38},"method":"PUT","path":"filtered","params":{"time":3815.904,"chapter_index":0},"response":[{}]},"node":"main","time":"2017-04-09T14:39:06UTC","tag":"filtered.console","@timestamp":"2017-04-09T23:39:06+09:00"}

问题是message 字段包含 JSON 对象，尽管该字段在 Elasticsearch 上被映射为分析字符串。

【讨论】：