如何使用fluentd解析带有json对象的日志答案

【问题标题】：how to parse log with json object using fluentd如何使用fluentd解析带有json对象的日志
【发布时间】：2019-12-17 05:20:03
【问题描述】：

我有一个 nginx 日志文件，其中包含如下行： 127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0" - <{"key1":"value1","key2":98765,"key3":false,"key4":["one","two"],"key5":{"key22":98765,"key23":false,"key24":["one-one","two-two"]}}> 如您所见，最后一个值是 JSON 对象。现在我需要按以下格式解析

1362020400 (28/Feb/2013:12:00:00 +0900)

record:
{
  "remote"              : "127.0.0.1",
  "host"                : "192.168.0.1",
  "user"                : "-",
  "method"              : "GET",
  "path"                : "/",
  "code"                : "200",
  "size"                : "777",
  "referer"             : "-",
  "agent"               : "Opera/12.0",
  "http_x_forwarded_for": "-",
  "myobject"            :{
                          "key1": "value1",
                          "key2": 98765,
                          "key3": false,
                          "key4": [
                            "one",
                            "two"
                          ],
                          "key5": {
                            "key22": 98765,
                            "key23": false,
                            "key24": [
                              "one-one",
                              "two-two"
                            ]
                          }
                         }

我可以使用这样的格式：

expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?) \<(?<myobject>[^\>]*)\>?$/
time_format %d/%b/%Y:%H:%M:%S %z

但这会将最后一个 JSON 对象解析为字符串。如何将该值保留为 JSON？

【问题讨论】：

标签： json regex parsing fluentd

【解决方案1】：

你可以使用filter/parser从字符串解析为json对象

<filter foo.bar>
  @type parser
  key_name myobject
  reserve_data true
  remove_key_name_field true
  hash_value_field parsed
  <parse>
    @type json
  </parse>
</filter>

这是一个例子

# input data:  {"host":"192.168.0.1", "myobject":"{\"key1\":1,\"key2\":2}"}
# output data: {"host":"192.168.0.1", "parsed":{"key1":1,"key2":2}}

【讨论】：