Logstash grok 过滤器调试

Question

请帮忙，我正在尝试在我的 Logstash 管道中添加 grok 过滤器，它将在 logline 下方转换

2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S

到

{
  "ts": [
    [
      "2020-11-06 12:57:43,854"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "11"
    ]
  ],
  "MONTHDAY": [
    [
      "06"
    ]
  ],
  "HOUR": [
    [
      "12",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "43,854"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "loglevel": [
    [
      "INFO"
    ]
  ],
  "Metric": [
    [
      "Bandwidth"
    ]
  ],
  "Chanel": [
    [
      "NASDAQ:224.0.130.65:30408"
    ]
  ],
  "Data": [
    [
      "0.000059 Gb/S"
    ]
  ]
}

下面是我的 grok 过滤器

input{
  beats{
    port => "5044"
  }
}

filter{
  if "Bandwidth" in [message]{
    grok{
      match => {"message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Chanel>[A-Z]+:[0-9]+.[0-9]+.[0-9]+.[0-9]+:[0-9]+)"}
    }
  }
}

output{
  elasticsearch{
    hosts => [ "localhost:9200" ]
  }
}

当我在 Grok 调试器中尝试时，这个过滤器工作得非常好，但在 Kibana 中查看时，在 Logstash 中却不行。我没有从过滤器中看到任何名称捕获。只是消息。如果我删除过滤器的正则表达式部分并添加 GREEDYDATA，一切正常。我确定我在正则表达式部分做错了什么。

Answer 1

您的正则表达式模式是正确的，并且确实提供了预期的过滤器输出。在 Kibana 中刷新您的索引模式或尝试重新获取数据。

不过，如果通道模式类似于 Some Data:IP Address:Port，我认为您不需要使用正则表达式

试试下面的模式

grok{
      match => { "message" => ["%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Channel>%{DATA}:%{HOSTPORT}) (?<Data>%{GREEDYDATA})"]}
    }

Logstash 输出将是

{
            "ts" => "2020-11-06 12:57:43,854",
        "Metric" => "Bandwidth",
    "@timestamp" => 2020-11-06T22:47:20.383Z,
      "loglevel" => "INFO",
          "host" => "e7c15acec470",
          "Data" => "0.000059 Gb/S",
       "Channel" => "NASDAQ:224.0.130.65:30408",
      "@version" => "1",
       "message" => "2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S"
}

尝试将 stdout 输出与 elasticsearch 一起使用，这样您就可以看到 logstash 输出到 elastic 的内容。

output{
      stdout { codec => rubydebug }
    }