如何更改索引中的“消息”值答案

【问题标题】：How to change “message” value in index如何更改索引中的“消息”值
【发布时间】：2022-01-20 16:19:56
【问题描述】：

在logstash管道或索引模式中，如何在“消息”字段中更改CDN日志的以下部分以分离或提取一些数据然后聚合它们。

<40> 2022-01-17T08:31:22Z logserver-5 testcdn[1]: {"method":"GET","scheme":"https","domain":"www.123.com","uri":"/product/10809350","ip":"66.249.65.174","ua":"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","country":"US","asn":15169,"content_type":"text/html; charset=utf-8","status":200,"server_port":443,"bytes_sent":1892,"bytes_received":1371,"upstream_time":0.804,"cache":"MISS","request_id":"b017d78db4652036250148216b0a290c"}

预期变化：

{"method":"GET","scheme":"https","domain":"www.123.com","uri":"/product/10809350","ip":"66.249.65.174","ua":"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)","country":"US","asn":15169,"content_type":"text/html; charset=utf-8","status":200,"server_port":443,"bytes_sent":1892,"bytes_received":1371,"upstream_time":0.804,"cache":"MISS","request_id":"b017d78db4652036250148216b0a290c"}

因为这部分“ 2022-01-17T08:31:22Z logserver-5 testcdn[1]:”没有在 jason 中解析，我无法创建基于一些文件，如国家、asn 等...

logstash 索引的原始日志是：

{
  "_index": "logstash-2022.01.17-000001",
  "_type": "_doc",
  "_id": "Qx8pZ34BhloLEkDviGxe",
  "_version": 1,
  "_score": 1,
  "_source": {
    "message": "<40> 2022-01-17T08:31:22Z logserver-5 testcdn[1]: {\"method\":\"GET\",\"scheme\":\"https\",\"domain\":\"www.123.com\",\"uri\":\"/product/10809350\",\"ip\":\"66.249.65.174\",\"ua\":\"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\",\"country\":\"US\",\"asn\":15169,\"content_type\":\"text/html; charset=utf-8\",\"status\":200,\"server_port\":443,\"bytes_sent\":1892,\"bytes_received\":1371,\"upstream_time\":0.804,\"cache\":\"MISS\",\"request_id\":\"b017d78db4652036250148216b0a290c\"}",
    "port": 39278,
    "@timestamp": "2022-01-17T08:31:22.100Z",
    "@version": "1",
    "host": "93.115.150.121"
  },
  "fields": {
    "@timestamp": [
      "2022-01-17T08:31:22.100Z"
    ],
    "port": [
      39278
    ],
    "@version": [
      "1"
    ],
    "host": [
      "93.115.150.121"
    ],
    "message": [
      "<40> 2022-01-17T08:31:22Z logserver-5 testcdn[1]: {\"method\":\"GET\",\"scheme\":\"https\",\"domain\":\"www.123.com\",\"uri\":\"/product/10809350\",\"ip\":\"66.249.65.174\",\"ua\":\"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\",\"country\":\"US\",\"asn\":15169,\"content_type\":\"text/html; charset=utf-8\",\"status\":200,\"server_port\":443,\"bytes_sent\":1892,\"bytes_received\":1371,\"upstream_time\":0.804,\"cache\":\"MISS\",\"request_id\":\"b017d78db4652036250148216b0a290c\"}"
    ],
    "host.keyword": [
      "93.115.150.121"
    ]
  }
}

谢谢

【问题讨论】：

标签： logstash kibana elastic-stack cdn elk

【解决方案1】：

谢谢，这很有用，我从你对这个特定场景的建议中得到了一个想法：以下编辑的 logstash.conf 解决了这个问题：

input {
  tcp {
        port => 5000
        codec => json
  }
}

filter {
   grok {
            match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:Junk}: %{GREEDYDATA:request}"}
        }
   json { source => "request" }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    manage_template => false
    ecs_compatibility => disabled
    index => "logs-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

但我主要关心的是编辑配置文件，我宁愿在 kibana web ui 中进行任何更改，而不是更改 logstash.conf，因为我们将 elk 用于组织中的不同场景，这样的更改使 elk 服务器适合只是一个特殊用途，而不是多用途。如何在不更改 logstash 配置文件的情况下获得这样的结果？

【讨论】：

我理解您的担忧。但这是标准 Elastic-stack 中的日志处理流程。根据您的要求，使用 kibana 将日志解析为单独字段的另一种方法是使用运行时字段。当您无权访问 logstash 服务器或者您不想更改 logstash 配置时，这会很有帮助。这有其优点和缺点，因为运行时字段是在从 kibana 查询时生成的，实际上并没有存储在任何地方。你可以在这里阅读更多内容：elastic.co/guide/en/elasticsearch/reference/7.16/runtime.html

【解决方案2】：

将这些配置添加到您的 logstash 配置的过滤部分：

#To parse the message field
grok {
    match => { "message" => "<%{NONNEGINT:syslog_pri}>\s+%{TIMESTAMP_ISO8601:syslog_timestamp}\s+%{DATA:sys_host}\s+%{NOTSPACE:sys_module}\s+%{GREEDYDATA:syslog_message}"}
}
#To replace message field with syslog_message
mutate {
    replace => [ "message", "%{syslog_message}" ]
}

将消息字段替换为 syslog_message 后，您可以添加下面的 json 过滤器以将 json 解析为单独的字段。..

json {
    source => "syslog_message"
}

【讨论】：