在logstash中过滤json答案

【问题标题】：filter json in logstash在logstash中过滤json
【发布时间】：2015-05-03 03:59:10
【问题描述】：

我有一个包含类似记录的 json 文件

{"id":1,"first_name":"Frank","last_name":"Mills","date":"5/31/2014","email":"fmills0@feedburner.com","country":"France","city":"La Rochelle","latitude":"46.1667","longitude":"-1.15"

我正在尝试过滤 logstash 中的字段，但到目前为止没有成功。我尝试了grok debugger 和grokconstructor，但无法使其工作。我的最后一次尝试是

input {
    file{
        path => ["C:/logstash-1.4.2/mock_data.json"]
        type => "json"
        start_position => "beginning"
        sincedb_path => "/dev/null"
  }
}
filter {
  mutate {
    replace => [ "message", "%{message}" ]
  }
  json {
    source => "message"
    remove_field => "message"
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", "longitude", "[location][lon]" ]
  }
}

output {
  stdout {
    codec => rubydebug
  } 
  elasticsearch {
    host => "127.0.0.1"
    protocol => "http"
    index => "test35"
  }
}

仅用于纬度和经度，但这不起作用。任何关于 Json 的 logstash 教程尤其如此。对此有任何帮助。具体配置文件的输出是

{
 "message" => "{\"id\":91,\"first_name\":\"Adam\",\"last_name\":\"Carr\",\"date\":\"11/14/2014\",\"email\":\"acarr2i@tinyurl.
com\",\"country\":\"Ghana\",\"city\":\"Mampong\",\"latitude\":\"7.06273\",\"longitude\":\"-1.4001\"},",
      "@version" => "1",
      "@timestamp" => "2015-05-04T19:05:08.409Z",
       "host" => "Toshiba",
       "path" => "C:/logstash-1.4.2/mock_data.json",
        "tags" => [
             [0] "_jsonparsefailure"
    ]
}

为 Alcanzar 更新

【问题讨论】：

标签： json logstash kibana elasticsearch-plugin

【解决方案1】：

geoip 过滤器用于将 IP 地址的纬度/经度添加到您的数据中。

将所有部分放在一起会产生这样的结果：

filter {
  grok {
        match => [ 'message', '(?<body>\"id\":.*\"longitude\":\"[^"]+\")' ]
        add_field => [ "json_body", "{%{body}}" ]
  }
  json {
        source => "json_body"
        remove_field => ["message","body","json_body" ]
  }
  mutate {
    convert => [ "latitude", "float" ]
    convert => [ "longitude","float" ]
  }
  mutate {
     rename => [ "latitude", "[location][lat]", 
       "longitude", "[location][lon]" ]
  }
}

这将生成如下所示的事件：

{
      "@version" => "1",
    "@timestamp" => "2015-05-04T19:48:52.051Z",
          "host" => "xxxxxxxx",
            "id" => 1,
    "first_name" => "Frank",
     "last_name" => "Mills",
          "date" => "5/31/2014",
         "email" => "fmills0@feedburner.com",
       "country" => "France",
          "city" => "La Rochelle",
      "location" => {
        "lat" => 46.1667,
        "lon" => -1.15
    }
}

这应该正是你想要的。

【讨论】：

我更新了我的问题。每次我尝试使用不同的索引。谢谢Alcanzar
我删除它并添加到输入编解码器 => json 因为我得到了 "tags" => [ [0] "_jsonparsefailure" ], "location" => [ [0] 0.0, [1] 0.0]
我的输入是准确的，没有遗漏的}。我也试过 mutate { replace => ["message","%{message}}"] } 但它仍然不起作用。
在我的机器上测试了完整的答案，它生成了应该工作的内容。
在主机之后我得到 "path" => "C:/logstash-1.4.2/mock_data.json", "tags" => [ [0] "_jsonparsefailure" ] } 我不不要像你一样得到数据。你能告诉我你使用的ELK版本吗？我使用 Elasticsearch 1.5.0、logstash 1.4.2 和 Kibana 4.0.1