为 logstash 使用多个配置文件答案

【问题标题】：Using multiple config files for logstash为 logstash 使用多个配置文件
【发布时间】：2023-04-01 12:25:01
【问题描述】：

我刚刚学习elasticsearch，我需要知道如何正确地将配置文件拆分为多个。我正在使用官方的logstash on docker，端口绑定在9600 和5044。最初我有一个没有条件的单个 logstash 文件，如下所示：

input {
    beats {
        port => '5044'
    }
}

filter
{
    grok{
        match => {
            "message" => "%{TIMESTAMP_ISO8601:timestamp} \[(?<event_source>[\w\s]+)\]:\[(?<log_type>[\w\s]+)\]:\[(?<id>\d+)\] %{GREEDYDATA:details}"
            "source" => "%{GREEDYDATA}\\%{GREEDYDATA:app}.log"
        }
    }
    mutate{
        convert => { "id" => "integer" }
    }
    date {
        match => [ "timestamp", "ISO8601" ]
        locale => en
        remove_field => "timestamp"
    }
}


output
{
    elasticsearch {
        hosts => ["http://elastic:9200"]
        index => "logstash-supportworks"
    }

}

当我想添加metricbeat 时，我决定将该配置拆分为一个新文件。所以我最终得到了 3 个文件：

__input.conf

input {
    beats {
        port => '5044'
    }
}

metric.conf

# for testing I'm adding no filters just to see what the data looks like

output {
  if ['@metadata']['beat'] == 'metricbeat' {
    elasticsearch {
        hosts => ["http://elastic:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
    }
  }
}

supportworks.conf

filter
{
    if ["source"] =~ /Supportwork Server/ {
        grok{
            match => {
                "message" => "%{TIMESTAMP_ISO8601:timestamp} \[(?<event_source>[\w\s]+)\]:\[(?<log_type>[\w\s]+)\]:\[(?<id>\d+)\] %{GREEDYDATA:details}"
                "source" => "%{GREEDYDATA}\\%{GREEDYDATA:app}.log"
            }
        }
        mutate{
            convert => { "id" => "integer" }
        }
        date {
            match => [ "timestamp", "ISO8601" ]
            locale => en
            remove_field => "timestamp"
        }
    }
}


output
{
    if ["source"] =~ /Supportwork Server/ {
        elasticsearch {
            hosts => ["http://elastic:9200"]
            index => "logstash-supportworks"
        }
    }

}

现在没有数据被发送到 ES 实例。我已经验证了 filebeat 至少正在运行并发布消息，所以我希望至少看到 ES 有这么多。这是我运行 filebeat 的服务器发布的消息

2019-03-06T09:16:44.634-0800    DEBUG   [publish]       pipeline/processor.go:308       Publish event: {
  "@timestamp": "2019-03-06T17:16:44.634Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.6.1"
  },
  "source": "C:\\Program Files (x86)\\Hornbill\\Supportworks Server\\log\\swserver.log",
  "offset": 4773212,
  "log": {
    "file": {
      "path": "C:\\Program Files (x86)\\Hornbill\\Supportworks Server\\log\\swserver.log"
    }
  },
  "message": "2019-03-06 09:16:42 [COMMS]:[INFO ]:[4924] Helpdesk API (5005) Socket error while idle - 10053",
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  },
  "beat": {
    "name": "WIN-22VRRIEO8LM",
    "hostname": "WIN-22VRRIEO8LM",
    "version": "6.6.1"
  },
  "host": {
    "name": "WIN-22VRRIEO8LM",
    "architecture": "x86_64",
    "os": {
      "platform": "windows",
      "version": "6.3",
      "family": "windows",
      "name": "Windows Server 2012 R2 Standard",
      "build": "9600.0"
    },
    "id": "e5887ac2-6fbf-45ef-998d-e40437066f56"
  }
}

【问题讨论】：

标签： elasticsearch logstash

【解决方案1】：

我通过向 __input.conf 添加一个 mutate 过滤器来将源字段中的反斜杠替换为正斜杠来实现此功能

filter {
    mutate{
        gsub => [ "source", "[\\]", "/" ]
    }
}

并从我的条件句中的字段访问器中删除"

if ["source"] =~ /Supportwork Server/

成为

if [source] =~ /Supportwork Server/

这两项更改似乎都是使此配置正常工作所必需的。

【讨论】：