logstash 解析旧日志现在添加时间戳和@timestamp 作为日期答案

【问题标题】：logstash parsing old logs adds timestamp and @timestamp as date nowlogstash 解析旧日志现在添加时间戳和@timestamp 作为日期
【发布时间】：2017-10-06 09:09:43
【问题描述】：

你好，我正在尝试解析一个旧的 apache 日志，输出有一个正确的时间戳，还有一个 @timestamp 字段，@timestamp 是现在的日期时间，我怎样才能确保时间戳成为 kibana 的 @timestamp /弹性搜索。示例输入：

172.31.21.26 - - [20/Jul/2017:22:1``0:52 +0200] "GET /mobile/getParent/NzE4MzU1ZmUtNmIwOC00N2JkLTk1YmYtNmNhZTUyZmVmNGYz HTTP/1.1" 200 452 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 (4301339520)"

配置文件：

input {
file {
      path=>"/home/ronald/Downloads/log/httpd/short.log"
      start_position=>"beginning"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" } 
    } 
}
output {
  elasticsearch{
    hosts=>"localhost"
    index=>"roha_test"
    document_type=>"demo1"
  }
  stdout{
      codec => "rubydebug"
  }
}

输出：

      "request" =>"/mobile/getParent/NzE4MzU1ZmUtNmIwOC00N2JkLTk1YmYtNmNhZTUyZmVmNGYz",
      "agent" => "\"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 (4301339520)\"",
       "auth" => "-",
      "ident" => "-",
       "verb" => "GET",
    "message" => "172.31.21.26 - - [20/Jul/2017:22:10:52 +0200] \"GET /mobile/getParent/NzE4MzU1ZmUtNmIwOC00N2JkLTk1YmYtNmNhZTUyZmVmNGYz HTTP/1.1\" 200 452 \"-\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Mobile/14G60 (4301339520)\"",
       "path" => "/home/ronald/Downloads/log/httpd/short.log",
   "referrer" => "\"-\"",
 "@timestamp" => 2017-10-06T08:49:10.440Z,
   "response" => "200",
      "bytes" => "452",
   "clientip" => "172.31.21.26",
   "@version" => "1",
       "host" => "ronald-XPS-13-9343",
"httpversion" => "1.1",
  "timestamp" => "20/Jul/2017:22:10:52 +0200"

logstash 版本 5.6.1

【问题讨论】：

添加date 过滤器。

标签： apache logging logstash

【解决方案1】：

您必须添加一个date filter，它将timestamp 字段转换为elasticsearch 可以理解的已解析日期时间对象。比如：

date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}

【讨论】：