Docker-compose 弹性栈无容器标签

Question

我有一个带有 docker-compose 和弹性堆栈的设置。我的“主”容器正在运行一个 Django 应用程序（还有一些用于度量、证书等的容器）。

日志记录本身适用于此设置，但我在 Kibana 中没有容器标签或标签。所以我无法区分来自不同容器的日志（除非我知道我在寻找什么）。

如何配置 logstash 或 logspout 以使用它们来自的容器标记或标记所有日志？在最好的情况下标记容器图像和容器 id。

我尝试为容器添加标签，但这并没有改变任何东西。我还尝试了使用驱动程序 syslog 和标签指定的日志记录，但这也不起作用。

我想我必须制作一个特定的 logstash 配置并在那里做一些事情？

下面是我当前的 docker-compose.yml

version: '2'

services:
    # django container
    web:
        build: .
        command: gunicorn backend.wsgi:application --bind 0.0.0.0:8001 --log-level debug
        restart: unless-stopped
        container_name: web
        depends_on:
            - logspout
        expose:
            - 8001
        env_file:
            - ./environments/web.test.env
        image: mycontainer
        labels:
            container: "web"
            com.example.service: "web"

    logspout:
        image: gliderlabs/logspout:v3.2.11
        command: 'udp://logstash:5000'
        restart: unless-stopped
        links:
            - logstash
        volumes:
            - '/var/run/docker.sock:/tmp/docker.sock'
        depends_on:
            - elasticsearch
            - logstash
            - kibana
        
    logstash:
        image: logstash:7.9.1
        restart: unless-stopped
        environment:
        - STDOUT=true
        links:
        - elasticsearch
        expose:
        - 5000
        depends_on:
        - elasticsearch
        - kibana
        command: 'logstash -e "input { udp { port => 5000 } } output { elasticsearch { hosts => elasticsearch } }"'

    kibana:
        image: kibana:7.9.1
        restart: unless-stopped
        links:
        - elasticsearch
        environment:
        - ELASTICSEARCH_URL=http://elasticsearch:9200
        ports:
        - 5601:5601
        depends_on:
        - elasticsearch

    elasticsearch:
        image: elasticsearch:7.9.1
        restart: unless-stopped
        ports:
        - 9200:9200
        - 9300:9300
        environment:
            - node.name=elasticsearch
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
            - cluster.initial_master_nodes=elasticsearch

任何帮助将不胜感激，谢谢！

Answer 1

对不起，我对弹性堆栈真的很缺乏经验，但我做对了。

确实，您必须提供带有过滤器的 logstash 配置，至少我是这样工作的。此外，我不得不在 logspout 中从 UDP 切换到仅 syslog，我猜 udp 连接没有转发它得到的所有内容（例如 docker 映像）。

这是我的有效配置（肯定有一些改进要做）。

logstash.conf

input {
  syslog {
      port => 5000
      type => "docker"
  }
}

filter {
  grok {
    match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:ver} +(?:%{TIMESTAMP_ISO8601:ts}|-) +(?:%{HOSTNAME:service}|-) +(?:%{NOTSPACE:containerName}|-) +(?:%{NOTSPACE:proc}|-) +(?:%{WORD:msgid}|-) +(?:%{SYSLOG5424SD:sd}|-|) +%{GREEDYDATA:msg}" }
  }
  syslog_pri { }
}

output {
  elasticsearch { hosts => "elasticsearch" }
  stdout {codec => rubydebug}
}

docker-compose.yml

version: '2'

services:
    web:
        build: .
        command: gunicorn backend.wsgi:application --bind 0.0.0.0:8001 --log-level debug
        restart: unless-stopped
        container_name: web
        depends_on:
            - logspout
        image: myimage
        expose:
            - 8001
        env_file:
            - ./environments/web.test.env

        labels:
            container: "web"
            com.example.service: "web"

    logspout:
        image: gliderlabs/logspout:v3.2.11
        command: 'syslog://logstash:5000'
        restart: unless-stopped
        links:
            - logstash
        volumes:
            - '/var/run/docker.sock:/tmp/docker.sock'
        depends_on:
            - elasticsearch
            - logstash
            - kibana
        
    logstash:
        image: logstash:7.9.1
        restart: unless-stopped
        environment:
            - LOGSPOUT=ignore
        links:
            - elasticsearch
        depends_on:
            - elasticsearch
            - kibana
        volumes:
            - ./containers/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf

    kibana:
        image: kibana:7.9.1
        restart: unless-stopped
        links:
            - elasticsearch
        environment:
            - LOGSPOUT=ignore
            - ELASTICSEARCH_URL=http://elasticsearch:9200
        ports:
            - 5601:5601
        depends_on:
            - elasticsearch

    elasticsearch:
        image: elasticsearch:7.9.1
        restart: unless-stopped
        ports:
        - 9200:9200
        - 9300:9300
        environment:
            - LOGSPOUT=ignore
            - node.name=elasticsearch
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
            - cluster.initial_master_nodes=elasticsearch