在 logstash 中使用 grock 将 nginx 日志转换为 elasticsearch答案

【问题标题】：convert nginx logs to elasticsearch using grock in logstash在 logstash 中使用 grock 将 nginx 日志转换为 elasticsearch
【发布时间】：2014-07-15 20:23:36
【问题描述】：

我从 nginx 收到以下消息：

XXX.XXX.XXX.XXX - - [09/Jul/2014:15:23:51 +0200] "GET /js/order.js HTTP/1.1" 200 2777 "http://www.yyy.xxx.zz /accueil"; "Mozilla/5.0 (Windows NT 5.1; rv:30.0) Gecko/20100101 Firefox/30.0" "-" "0.000" "317"

我需要将数据收集到elasticsearch文档：

httpversion 1.1
method      GET
offset      127904969
request     /js/order.js
response    200
timestamp   09/Jul/2014:15:23:51 +0200
type        nginx
url         http://www.geoportail.gouv.fr/accueil

为了实现这一点，我使用了以下模式：

match => [ "message", "%{IPORHOST:clientip} - - \[%{log-date:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]

我遇到的问题是所有字段都被视为字符串：

"method"   :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"offset"   :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"request"  :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"response" :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"timestamp":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"type"     :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}},
"url"      :{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}

我想将时间戳字段设置为“日期”甚至更好，用这个时间戳替换@timestamp，以便能够使用elasticsearch的范围api查询它。

按照 Alcanzar 的建议，我修改了 conf 文件如下：

input {
  lumberjack {
    # The port to listen on                                                                                                                                  
    port => 5140

    # The paths to your ssl cert and key                                                                                                                     
    ssl_certificate => "/etc/logstash/logstash.crt"
    ssl_key => "/etc/logstash/logstash.key"

    # Set this to whatever you want.                                                                                                                         
    type => "nginx"
  }
}

filter {
  grok {
    match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\
:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]
  }
}

filter {
 date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] }
}

output {
  elasticsearch {
    host => localhost
    index => front_gpp3
  }
  stdout { codec => rubydebug }
}

问题是语言环境，我的 VM 将 FR 作为语言环境而不是 EN，所以我添加以将其添加到过滤器中：

filter {
  grok {
    match => [ "message", "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER\
:response} (?:%{NUMBER:bytes}|-) \"%{URI:url}" ]
  }
  date {
    locale => "en"
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

【问题讨论】：

标签： logstash

【解决方案1】：

您需要添加一个date 过滤器：

filter {
 date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] }
}

或类似的东西。这将使用该格式解析timestamp，然后使用解析后的值设置@timestamp。

【讨论】：