如何将 json 完全解析为 ElasticSearch？答案

【问题标题】：How can I fully parse json into ElasticSearch?如何将 json 完全解析为 ElasticSearch？
【发布时间】：2021-11-23 14:11:10
【问题描述】：

我正在解析一个mongodb输入到logstash，配置文件如下：

input {
    mongodb {
        uri => "<mongouri>"
        placeholder_db_dir => "<path>"
        collection => "modules"
        batch_size => 5000
    }
}
filter {
        mutate  {
            rename => { "_id" => "mongo_id" }
            remove_field => ["host", "@version"]
        }
        json   {
            source => "message"
            target => "log"
        } 
}
output {
        stdout {
            codec => rubydebug
        }
        elasticsearch {
            hosts => ["localhost:9200"]
            action => "index"
            index => "mongo_log_modules"
        }
}

将集合中的 2/3 文档输出到 elasticsearch。

{
    "mongo_title" => "user",
      "log_entry" => "{\"_id\"=>BSON::ObjectId('60db49309fbbf53f5dd96619'), \"title\"=>\"user\", \"modules\"=>[{\"module\"=>\"user-dashboard\", \"description\"=>\"User Dashborad\"}, {\"module\"=>\"user-assessment\", \"description\"=>\"User assessment\"}, {\"module\"=>\"user-projects\", \"description\"=>\"User projects\"}]}",
       "mongo_id" => "60db49309fbbf53f5dd96619",
        "logdate" => "2021-06-29T16:24:16+00:00",
    "application" => "mongo-modules",
     "@timestamp" => 2021-10-02T05:08:38.091Z
}
{
    "mongo_title" => "candidate",
      "log_entry" => "{\"_id\"=>BSON::ObjectId('60db49519fbbf53f5dd96644'), \"title\"=>\"candidate\", \"modules\"=>[{\"module\"=>\"candidate-dashboard\", \"description\"=>\"User Dashborad\"}, {\"module\"=>\"candidate-assessment\", \"description\"=>\"User assessment\"}]}",
       "mongo_id" => "60db49519fbbf53f5dd96644",
        "logdate" => "2021-06-29T16:24:49+00:00",
    "application" => "mongo-modules",
     "@timestamp" => 2021-10-02T05:08:38.155Z
}

似乎标准输出的输出将无法解析的代码抛出

“日志条目”

添加“重命名”字段后，“模块”不会添加字段。

我已经尝试过 grok mutate 过滤器，但在 _id %{DATA}、%{QUOTEDSTRING} 和 %{WORD} 之后对我不起作用。

我也尝试将嵌套映射更新到索引中，但似乎也不起作用

还有什么我可以尝试将完全嵌套的代码放入 elasticsearch 的方法吗？

【问题讨论】：

标签： elasticsearch

【解决方案1】：

解决办法是用 mutate 过滤

mutate { gsub => [ "log_entry", "=>", ": " ] }
mutate { gsub => [ "log_entry", "BSON::ObjectId\('([0-9a-z]+)'\)", '"\1"' ]}
json { source => "log_entry" remove_field => [ "log_entry" ] }

输出到标准输出

       "_id" => "60db49309fbbf53f5dd96619",
     "title" => "user",
   "modules" => [
    [0] {
             "module" => "user-dashboard",
        "description" => "User Dashborad"
    },
    [1] {
             "module" => "user-assessment",
        "description" => "User assessment"
    },
    [2] {
             "module" => "user-projects",
        "description" => "User projects"
    }
],

【讨论】：