logstash 解析 apache drupal 日志答案

【问题标题】：logstash parsing apache drupal logslogstash 解析 apache drupal 日志
【发布时间】：2020-07-22 01:35:49
【问题描述】：

我有这个日志：

[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company Care|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.15.19.140|uri=https://webdemo2nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=68055|link=|message=Failed to add card. HTTP Response: {"code":71,"category":1,"message":"Declined","reference":""}', referer: https://webdemo2.nics.com/ca/user/payment-information

我正在尝试通过 logstash 发送它。这是我的logstash配置：

input {
    file {
      path => "/tmp/test.log"
    }
}
filter {
  date {
    match => [ "logdate", "E MMM dd HH:mm:ss.SSS yyyy" ]
    target => "@timestamp"
  }
  grok {
        match => { "message" => "\[(?<timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s++%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri=%{URIPATH:uri}\|uid=%{NUMBER:uid}\|.*%{GREEDYDATA:message}$"
                 }
        overwrite => ["message"]
  }
  mutate{
      remove_field => "@version"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

问题在于它没有用时间戳覆盖@timestamp。我尝试使用 mutate rename 但这只是让我失去了这个领域。有谁知道如何更改配置文件来解决时间戳问题？此外，时间戳似乎被解析为字符串而不是日期对象。它应该来自我的日期模式还是日期模式和 grok 模式的组合？

这是输出：

{
        "client" => "154.15.19.140:64717",
    "error_code" => "AH01071",
    "@timestamp" => 2020-07-15T16:58:52.916Z,
      "loglevel" => "error",
          "host" => "kourosh-VirtualBox",
          "port" => "64717",
          "path" => "/tmp/test.log",
     "timestamp" => "Fri May 15 01:20:48.661420 2020",
       "message" => "[Fri May 15 01:20:48.661420 2020] [proxy_fcgi:error] [pid 11943:tid 139947462346496] [client 154.15.19.140:64717] AH01071: Got error 'PHP message: Company|https://webdemo2.nics.com|severity=error|type=feature_card_manager|ip=154.5.59.140|uri=https://webdemo2.nics.com/ca/user/payment-information|referer=https://webdemo2.nics.com/ca/user/payment-information|uid=55|link=|message=Failed to add card. HTTP Response: {\"code\":71,\"category\":1,\"message\":\"Declined\",\"reference\":\"\"}', referer: https://webdemo2.nics.com/ca/user/payment-information"
}

【问题讨论】：

标签： logstash

【解决方案1】：

我想通了。 date 部分需要在 grok 部分之后，并且 target 可用于覆盖 @timestamp。这是工作配置：

    input {
    file {
      path => "/tmp/test.log"
    }
}
filter {
  grok {
        match => { "message" => "\[(?<event_timestamp>%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR})\]\s+\[%{WORD}:%{WORD:loglevel}\].*\[client %{URIHOST:client}\]\s+%{WORD:error_code}\:.*\|type=%{WORD:type}\|.*\|uri\=%{GREEDYDATA:uri}\|referer.*\|uid=%{NUMBER:uid}\|.*\|message\=%{GREEDYDATA:message}(\s+HTTP Response\:%{GREEDYDATA:response}.*)?(\,\sreferer.*).*"
                 }
        overwrite => ["message"]
  }
  date {
    match => [ "event_timestamp", "E MMM dd HH:mm:ss.SSSSSS yyyy" ]
    target => "@timestamp"
    add_field => { "debug" => "timestampMatched"}
  }

  mutate{
      remove_field => ["@version", "@tags"]
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

【讨论】：