分布式跟踪和弹性堆栈可视化

Question

2019-06-03 10:45:00.051  INFO [currency-exchange,411a0496b048bcf4,8d40fcfea92613ad,true] 45648 --- [x-Controller-10] logger                                   : inside exchange

这是我控制台中的日志格式。我正在使用spring cloud stream将我的日志从应用程序传输到logstash。这是logstash中日志解析的格式

grok {
              match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" }
       }

这是我的logstash.conf

input { kafka { topics => ['zipkin'] } } filter { # pattern matching logback pattern grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" } } } output { elasticsearch {hosts => ['localhost:9200'] index => 'logging'} stdout {} }

这是我在 log-stash console 中的输出。这是解析异常

{ "消息" => "[{\"traceId\":\"411a0496b048bcf4\",\"parentId\":\"8d40fcfea92613ad\",\"id\":\"f14c1c332d2ef077\",\"kind\": \"CLIENT\",\"name\":\"get\",\"timestamp\":1559538900053889,\"duration\":16783,\"localEndpoint\":{\"serviceName\":\"currency -exchange\",\"ipv4\":\"10.8.0.7\"},\"tags\":{\"http.method\":\"GET\",\"http.path\":\ "/convert/1/to/4\"}},{\"traceId\":\"411a0496b048bcf4\",\"parentId\":\"411a0496b048bcf4\",\"id\":\"8d40fcfea92613ad\" ,\"name\":\"hystrix\",\"timestamp\":1559538900050039,\"duration\":34500,\"localEndpoint\":{\"serviceName\":\"currency-exchange\", \"ipv4\":\"10.8.0.7\"}},{\"traceId\":\"411a0496b048bcf4\",\"id\":\"411a0496b048bcf4\",\"kind\":\"SERVER \",\"名称\":\"获取 /convert\",\"timestamp\":1559538900041446,\"duration\":44670,\"localEndpoint\":{\"serviceName\":\"currency-exchange\",\"ipv4\":\" 10.8.0.7\"},\"remoteEndpoint\":{\"ipv6\":\"::1\",\"port\":62200},\"tags\":{\"http.method\ ":\"GET\",\"http.path\":\"/convert\",\"mvc.controller.class\":\"Controller\",\"mvc.controller.method\":\ “转变\”}}]”， "@timestamp" => 2019-06-03T05:15:00.296Z, "@version" => "1", “标签” => [ [0]“_grokparsefailure” ] }

Answer 1

当我使用 Kibana 中内置的 Grok 调试器（在开发工具下）时，我从您的示例日志和 grok 模式中得到以下结果：

{
  "severity": "DEBUG",
  "rest": "GET \"/convert/4/to/5\", parameters={}",
  "pid": "35973",
  "thread": "nio-9090-exec-1",
  "trace": "62132b44a444425e",
  "exportable": "true",
  "service": "currency-conversion",
  "class": "o.s.web.servlet.DispatcherServlet",
  "timestamp": "2019-05-31 05:31:42.667",
  "span": "62132b44a444425e"
}

这对我来说是正确的。那么缺失的部分是什么？

您显示的日志输出还包含"ipv4":"192.168.xx.xxx"},"remoteEndpoint": {"ipv6":"::1","port":55394},"tags": ...，它不在示例日志中。这是哪里来的？