Logstash 无法从 json 赋值答案

【问题标题】：Logstash unable to assign value from jsonLogstash 无法从 json 赋值
【发布时间】：2016-06-23 18:57:13
【问题描述】：

给定以下输入：

111.111.111.111, 11.11.11.11 - 11.11.11.11 [06/May/2016:08:26:10 +0000] "POST /some-service/GetSomething HTTP/1.1" 499 0 "-" "Jakarta Commons- HttpClient/3.1" "7979798797979799" 59.370 - "{\x0A\x22correlationId\x22 : \x22TestCorr1\x22\x0A}"

使用 Logstash 配置：

input { stdin {} }
output { stdout { codec => "rubydebug" } }

filter {
   grok {
     match => { "message" => "%{COMBINEDAPACHELOG} %{QS:partner_id} %{NUMBER:req_time} %{GREEDYDATA:extra_fields}" }
     add_field => [ "received_at", "%{@timestamp}" ]
     add_field => [ "received_from", "%{host}" ]
   }


   mutate {
     gsub => ["extra_fields", "\\x0A","","extra_fields", "\\x22",'\"']
   }

   json {
     source => "extra_fields"
     target => "extra_fields_json"
   }


    mutate {
      add_field => {
        "correlationId" => "%{[extra_fields_json][correlationId]}"
      }
   }
}

给了我们以下输出：

Logstash 启动完成 { "消息" => "111.111.111.111, 11.11.11.11 - 11.11.11.11 [06/May/2016:08:26:10 +0000] "POST /some-service/GetSomething HTTP/1.1" 499 0 "-" " Jakarta Commons-HttpClient/3.1" "8888888" 59.370 - "{\x0A\x22correlationId\x22 : \x22TestCorr1\x22\x0A}", "@version" => "1", "@timestamp" => "2016-06-23T18:00:28.831Z", “客户端” => “11.11.11.11”， “身份” => “-”， “身份验证” => “10.0.12.205”， "时间戳" => "06/May/2016:08:26:10 +0000", "动词" => "POST", "请求" => "/some-service/GetSomething", "httpversion" => "1.1", “响应” => “499”， “字节”=>“0”， "推荐人" => "\"-\"", "代理" => "\"Jakarta Commons-HttpClient/3.1\"", "partner_id" => "\"8888888\"", “请求时间”=> [ [0] "59.370", [1] “1” ], "extra_fields" => "\"{\\\"correlationId\\\" : \\\"TestCorr_1\\\"}\"", "extra_fields_json" => "{\"correlationId\" : \"TestCorr_1\"}", “correlationId” => “correlationId” } Logstash 关闭完成

为什么“correlationId”的值是“correlationId”而不是“TestCorr_1\”？？？

感谢您的帮助。谢谢！

【问题讨论】：

标签： logstash logstash-grok

【解决方案1】：

您必须从extra_fields 中删除额外的" 和\ 才能正确解析json。您必须将您的 mutate 过滤器更改为：

mutate {
      gsub => ["extra_fields", "\"","",
          "extra_fields", "\\x0A","",
          "extra_fields", "\\x22",'\"',
          "extra_fields", "(\\)",""
      ]
 }

如果有效/无效，请随时提供反馈

【讨论】：

太棒了...有效！非常感谢我花了几天时间来解决这个问题。顺便说一句，还有一个问题，如果 json 为空，它会显示文本，如果源没有字段 "correlationId" => "%{[extra_fields_json][correlationId]}" ，我该如何使其为空，应该是"correlationId" => "" 如果源中不存在该字段。
我认为你必须像这样使用 sthg，在 json 过滤器之后： if ([correlationId] == "%{[extra_fields_json][correlationId]}") {...} with mutate 过滤器和correlationId 字段上的replace 选项（doc）
嗨@yanto，如果答案已经解决了您的问题，请考虑接受它。如果您需要进一步的帮助，请随时发表评论/提出新问题