grok 解析多行，例如异常堆栈跟踪

Question

我的日志文件中有多种（三种）类型的日志。其中一种类型有一些自己的打印+异常堆栈跟踪。示例如下：

Multiple lines example:
    2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

我有一个模式来解析它，它是：

#pattern:
(?<timestamp>[\d\-\s\:]+)\s\[(?<threadname>[\w\-\d]+)\]\s-\s(?<loglevel>[\w]+)\s\-\s(?<appinfo>app-info-exception-info)\s-\s(?<params>params):(?<jsonstr>[\"\w\d\,\:\.\{\}]+)\s(?<exceptionname>[\w\d\.]+Exception):\s(?<exceptiondetail>[\w\d\.]+)\n\t(?<extralines>at[\s\w\.\d\~\?\n\t\(\)\_\[\]\/\:\-]+)\n

Pattern 在解析多行异常堆栈跟踪时有错误（实际上不是错误，但没有完全解析或按预期解析），主要在最后两部分（exceptiondetail（在这种情况下为空）和extralines（那些以空格或制表符开头的行）加上'at'，或堆栈跟踪第一行之后的行））。有比我更好的主意吗？

在 filebeat.yml 中，我进行了以下配置：

# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: '^[[:space:]]'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: false

  multiline.match: after

有什么想法可以改进多行解析（stacktrace 异常）？

Answer 1

如何让它更简单？使用 (?m) 将额外数据（所有行以 at 开头）分配给 GREEDYDATA 到单个字段中？

例如，如果这是您的日志，

2018-04-27 10:53:17 [http-nio-8088-exec-4] - ERROR - app-info-exception-info - params:{"cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"} java.lang.NullPointerException: null
        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]
        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

你可以把它解析为，

%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:threadname}\] - %{LOGLEVEL:loglevel} - app-info-exception-info - params:%{SPACE}\{\"%{DATA:jsondata}\"\} %{DATA:excentionname}: %{DATA:exceptiondetail}\n(?m)%{GREEDYDATA:extralines}

哪个会输出，

{
  "timestamp": [
    [
      "2018-04-27 10:53:17"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "04"
    ]
  ],
  "MONTHDAY": [
    [
      "27"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "53",
      null
    ]
  ],
  "SECOND": [
    [
      "17"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "threadname": [
    [
      "http-nio-8088-exec-4"
    ]
  ],
  "loglevel": [
    [
      "ERROR"
    ]
  ],
  "SPACE": [
    [
      ""
    ]
  ],
  "jsondata": [
    [
      "cardid":"111111111","txamt":10,"ip":"192.168.16.89","stationcode":"0002","inputuserid":1,"organcode":"99999"
    ]
  ],
  "excentionname": [
    [
      "java.lang.NullPointerException"
    ]
  ],
  "exceptiondetail": [
    [
      "null"
    ]
  ],
  "extralines": [
    [
      "        at com.datalook.group.BusinessHandler.handler(BusinessHandler.java:93) ~[classes/:?]\n        at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:51) [classes/:?]\n        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]\n        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]\n        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]\n        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]"
    ]
  ]
}

您可以将(?m) 替换为%{SPACE} 以将每行以at 开头的行也分成自己的字段。

Answer 2

我认为您可以这样做并告诉每个新行都以时间戳开头：

multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after