无法在 ELK 5.4 上正确解析时间戳答案

【问题标题】：unable to parse timetstamp correctly on ELK 5.4无法在 ELK 5.4 上正确解析时间戳
【发布时间】：2017-10-29 11:58:50
【问题描述】：

我正在使用最后一个 ELK 堆栈 (5.4.0)。我正在解析一些 apache 日志。 elasticsearch 2.4.5 和 kibana 4.6.4 一切正常

OK版

apache 173.252.115.89 - - [29/May/2017:09:59:13 +0200] "GET /fr/fia/nodes.rss HTTP/1.1" 200 19384 "-" "facebookexternalhit/1.1" "-" 756752 "*/*" monsite.com

使用以下 grok conf 完美输入 elasticsearch

grok {
          match => { "message" => "%{WORD:program} %{COMBINEDAPACHELOG} \"((?<x_forwarded_for>%{IP:xff_clientip}.*)|-)\" %{NUMBER:request_time:float} %{QUOTEDSTRING:accept} %{IPORHOST:targethost}"}
        }
        date {
           match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }

使用以下 kibana conf

问题

使用 ELK 5.4，我得到完全相同的消息（来自重复的 rabbitmq 队列）、相同的 logstash conf 和“全新安装”，但我得到了

弹性搜索日志

[2017-05-29T10:17:58,498][DEBUG][o.e.a.b.TransportShardBulkAction] [srv-elk-01] [logstash-2017.05.29][1] failed to execute bulk item (index) BulkShardRequest [[logstash-2017.05.29][1]] containing
 [16] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.4.0.jar:5.4.0]
...
at org.elasticsearch.index.mapper.DateFieldMapper.parseCreateField(DateFieldMapper.java:468) ~[elasticsearch-5.4.0.jar:5.4.0]
        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.4.0.jar:5.4.0]
        ... 40 more

logstash 日志

[2017-05-29T10:17:58,503][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.05.29", :_type=>"syslog", :_routing=>nil}, 2017-05-29T08:17:57.000Z 212.95.67.139 apache 212.95.70.118 - - [29/May/2017:10:17:57 +0200] "GET /de/tag/opera HTTP/1.1" 200 8948 "-" "TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)" "199.47.87.143, 199.47.87.143" 784504 "text/*,application/*" monsite.com], :response=>{"index"=>{"_index"=>"logstash-2017.05.29", "_type"=>"syslog", "_id"=>"AVxTSHzL1K94bfQE3eaM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"29/May/2017:10:17:57 +0200\" is malformed at \"/May/2017:10:17:57 +0200\""}}}}}

我的 kibana 配置是

【问题讨论】：

标签： elasticsearch logstash kibana elasticsearch-5 kibana-5

【解决方案1】：

Kibana 的最新版本不使用已弃用的时间戳字段。在您的 logstash conf 文件中使用“日期”。更多详情请看这里： timestamp

【讨论】：

【解决方案2】：

解决方案是删除时间戳，如下所示

date {
           match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
           remove_field => ["timestamp"]
        }

然后就没事了

【讨论】：