Logstash grok 配置错误 - 解析文件名时

Question

我是 logstash 和 grok 的新手，我正在尝试以以下格式解析 S3 存储桶中的 AWS ECS 日志 -

文件名 - my-logs-s3-bucket/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz

示例行 -

2019-05-09T16:16:16.983Z   JBoss Bootstrap Environment
2019-05-09T16:16:16.983Z   JBOSS_HOME: /app/jboss
2019-05-09T16:16:16.983Z   JAVA_OPTS:  -server -XX:+UseCompressedOops  -Djboss.server.log.dir=/var/log/jboss -Xms128m -Xmx4096m 

还有logstash.conf

input {
  s3 {
    region => "us-east-1"
    bucket => "my-logs-s3-bucket"
    interval => "7200"
  }
}

filter {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
  }
  date {
    match => ["tstamp", "ISO8601"]
  }
  mutate {
    remove_field => ["tstamp"]
    add_field => {
      "file" => "%{[@metadata][s3][key]}"
    }
    ######### NEED HELP HERE - START #########
    #grok {
    #  match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    #}
    ######### NEED HELP HERE - END #########
  }
}

output {
 stdout { codec => rubydebug {
    #metadata => true
  }
 }
}

当我使用上述配置运行 logstash 时，我能够看到所有解析的日志和提取的文件名，输出中的文件名如下所示 -

"file" => "myapp-logs/3d265ee3-d2ee-4029-a3d9-fd2255d69b92/ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842/000000.gz",

我正在尝试使用 grok 将文件名提取为 ecs-fargate-container-8ff0e472-c76f-4f61-a363-64c2b80aa842 或 8ff0e472-c76f-4f61-a363-64c2b80aa842，方法是取消注释 #NEED HELP HERE - START 之间的 grok 配置行并以以下错误结尾 -

Expected one of #, => at line 21, column 10 (byte 536) after filter {\n grok {\n match => [\"message\", \"%{TIMESTAMP_ISO8601:tstamp}\"]\n }\n date {\n match => [\"tstamp\", \"ISO8601\"]\n }\n mutate {\n #remove_field => [\"tstamp\"]\n add_field => {\n \"file\" => \"%{[@metadata][s3][key]}\"\n }\n grok ", :

我不确定我哪里出了问题。请指教。

Answer 1

您的grok 过滤器位于mutate 过滤器内，请尝试以下操作。

filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:tstamp}"]
    }
    date {
        match => ["tstamp", "ISO8601"]
    }
    mutate {
        remove_field => ["tstamp"]
        add_field => { "file" => "%{[@metadata][s3][key]}" }
    }
    grok {
        match => [ "file", "ecs-fargate-container-%{DATA:containerlogname}"]
    }
}