无法使用 logstash 将 .log 解析为 .json答案

【问题标题】：Can't parse .log to .json with logstash无法使用 logstash 将 .log 解析为 .json
【发布时间】：2019-01-31 13:36:40
【问题描述】：

我是 logstash 的新手，正在尝试将 .log 文件解析为 .json 在输出文件中有 "tags":["_grokparsefailure"] 和 "message": 包含所有 xml 文件

日志文件：

2019-01-18 14:03:07,666 - Request - ..................... - http://......................................................................................... - getOpenInvoices - 
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:tns="http://....................................................................." xmlns:types="http://............................................................................................" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
        <tns:getOpenInvoices>
            <invoiceQueryOpenRequest href="#id1" />
        </tns:getOpenInvoices>
        <q1:InvoiceQueryOpenRequest id="id1" xsi:type="q1:InvoiceQueryOpenRequest" xmlns:q1="java:com.....................collgw.model.invoice">
            <bankId xsi:type="xsd:int">23</bankId>
            <compId xsi:type="xsd:int">533</compId>
            <curr xsi:type="xsd:string">949</curr>
            <custId xsi:nil="true" />
            <invCount xsi:type="xsd:int">5</invCount>
            <msgDate xsi:nil="true" />
            <msisdn xsi:type="xsd:long">123456789</msisdn>
            <orig xsi:nil="true" />
            <period xsi:type="xsd:string">201901</period>
            <procDate xsi:nil="true" />
            <procTime xsi:nil="true" />
            <sessionId xsi:type="xsd:string">.............</sessionId>
            <stan xsi:type="xsd:long">0</stan>
        </q1:InvoiceQueryOpenRequest>
    </soap:Body>
</soap:Envelope>

配置文件：

input {
    file {
    path => "C:\Users............\Desktop\xml\20190118.log"
    type => "test-xml"
    start_position => "beginning"
    codec => multiline {
           pattern => "^"
           negate => true
          what => "previous"
         }
     }
}
filter {
    xml {
      store_xml => "false"
      source => "data"
      xpath => [
"/soap:Envelope/soap:Body/q1:InvoiceQueryOpenRequest/bankId/text()", "bankId",
"/soap:Envelope/soap:Body/q1:InvoiceQueryOpenRequest/compId/text()", "compId"
       ]
     }
mutate {
rename => [
"[bankId][0]", "bankId",
"[compId][0]", "compId"
       ]
     }
  }

output {
  file
     {
       path => "C:\Users............\Desktop\xml2\20190118.json"
     }
}

问题可能是过滤器问题。 BankId 或 compId key ，都在消息键中。

【问题讨论】：

为什么你有一个 grok 过滤器？这是无用的，因为此过滤器设置的两个字段都将被 xml 过滤器覆盖。

标签： json xml logstash

【解决方案1】：

您可以在mutate 中使用gsub。检查下面的示例：

mutate { gsub => [ "message", "^[^<]+<", "<" ] } xml { source => "message" target => "theXML" store_xml => true } 
filter {

    mutate { gsub => [ "message", "^[^<]+<", "<" ] }
    xml {
        source => "message"
        target => "theXML"
        store_xml => true
        xpath =>{
                        "//q1:InvoiceQueryOpenRequest/*[last()]" => "nvoiceQueryOpenRequest"

                }
    }
    mutate {
      remove_field => ["message",
                       "[theXML]"
                      ]
    }

}

【讨论】：