通过kafka使用logstash进行无限日志记录

Question

概述

我正在尝试使用 docker 容器让 Logstash 与 Kafka 一起工作。

对于我的 Kafka 容器，我使用的是 spotify build。我也在使用the official Logstash container。

我想实现以下目标：

这是我用于 Logstash 的配置：

file_kafka.conf

input {
    file {
        path => ["/data/*.log"]
        start_position => "beginning"
    }
}

output {
    kafka {
        topic_id => "test"
        bootstrap_servers => "<host>:9092"
    }
}

kafka_stdout.conf

input {
    kafka {
        topics => ["test"]
        bootstrap_servers => "<host>:9092"
        auto_offset_reset => "latest"
    }
}

output {
    stdout {}
}

这个设置对我来说似乎完全合理。事实上，它一开始确实奏效了。

问题

我启动了相关的容器，Logstash 正确地拾取了我的 test.log 文件中的单行并将其打印到屏幕上。

但是，我随后运行了以下命令：

echo 'blah' >> test.log

我从 logstash 收到了这个输出：

2017-05-11T18:36:49.090Z %{host} 2017-05-11T18:36:49.082Z %{host} 2017-05-11T18:36:49.076Z %{host} 2017-05-11T18:36:49.069Z %{host} 2017-05-11T18:36:49.062Z %{host} 2017-05-11T18:36:49.053Z %{host} 2017-05-11T18:36:49.044Z %{host} 2017-05-11T18:36:49.034Z %{host} 2017-05-11T18:36:49.025Z %{host} 2017-05-11T18:36:49.016Z %{host} 2017-05-11T18:36:49.009Z %{host} 2017-05-11T18:36:49.003Z %{host} 2017-05-11T18:36:48.995Z %{host} 2017-05-11T18:36:48.988Z %{host} 2017-05-11T18:36:48.982Z %{host} 2017-05-11T18:36:48.975Z %{host} 2017-05-11T18:36:48.968Z %{host} 2017-05-11T18:36:48.948Z %{host} 2017-05-11T18:36:48.927Z %{host} 2017-05-11T18:36:48.911Z %{host} 2017-05-11T18:36:48.896Z %{host} 2017-05-11T18:36:48.881Z %{host} 2017-05-11T18:36:48.868Z %{host} 2017-05-11T18:36:48.857Z %{host} 2017-05-11T18:36:48.849Z %{host} 2017-05-11T18:36:48.841Z %{host} 2017-05-11T18:36:48.830Z %{host} 2017-05-11T18:36:48.820Z %{host} 2017-05-11T18:36:48.805Z %{host} 2017-05-11T18:36:48.791Z %{host} 2017-05-11T18:36:48.779Z %{host} 2017-05-11T18:36:48.492Z %{host} 2017-05-11T18:36:48.483Z %{host} 2017-05-11T18:36:48.471Z %{host} 2017-05-11T18:36:48.463Z %{host} 2017-05-11T18:36:48.454Z %{host} 2017-05-11T18:36:48.445Z %{host} 2017-05-11T18:36:48.437Z %{host} 2017-05-11T18:36:48.427Z %{host} 2017-05-11T18:36:48.420Z %{host} 2017-05-11T18:36:48.412Z %{host} 2017-05-11T18:36:48.406Z %{host} 2017-05-11T18:36:48.399Z %{host} 2017-05-11T18:36:48.391Z %{host} 2017-05-11T18:36:48.385Z %{host} 2017-05-11T18:36:48.378Z %{host} 2017-05-11T18:36:48.371Z %{host} 2017-05-11T18:36:48.356Z %{host} 2017-05-11T18:36:48.286Z %{host} 2017-05-11T18:36:48.278Z %{host} 2017-05-11T18:36:48.269Z %{host} 2017-05-11T18:36:48.261Z %{host} 2017-05-11T18:36:48.246Z %{host} 2017-05-11T18:36:48.238Z %{host} 2017-05-11T18:36:48.229Z %{host} 2017-05-11T18:36:48.219Z %{host} 2017-05-11T18:36:48.209Z %{host} 2017-05-11T18:36:48.201Z %{host} 2017-05-11T18:36:48.191Z %{host} 2017-05-11T18:36:48.182Z %{host} 2017-05-11T18:36:48.175Z %{host} 2017-05-11T18:36:48.167Z %{host} 2017-05-11T18:36:48.112Z %{host} 2017-05-11T18:36:48.103Z %{host} 2017-05-11T18:36:48.095Z %{host} 2017-05-11T18:36:48.086Z %{host} 2017-05-11T18:36:48.077Z %{host} 2017-05-11T18:36:48.068Z %{host} 2017-05-11T18:36:48.059Z %{host} 2017-05-11T18:36:48.048Z %{host} 2017-05-11T18:36:48.031Z %{host} 2017-05-11T18:36:48.016Z %{host} 2017-05-11T18:36:48.009Z %{host} 2017-05-11T18:36:48.002Z %{host} 2017-05-11T18:36:47.994Z %{host} 2017-05-11T18:36:47.986Z %{host} 2017-05-11T18:36:47.972Z %{host} 2017-05-11T18:36:47.947Z %{host} 2017-05-11T18:36:47.923Z %{host} 2017-05-11T18:36:47.904Z %{host} 2017-05-11T18:36:47.879Z %{host} 2017-05-11T18:36:47.864Z %{host} 2017-05-11T18:36:47.853Z %{host} 2017-05-11T18:36:47.842Z %{host} 2017-05-11T18:36:47.834Z %{host} 2017-05-11T18:36:47.820Z %{host} 2017-05-11T18:36:47.810Z %{host} 2017-05-11T18:36:47.799Z %{host} 2017-05-11T18:36:47.792Z %{host} 2017-05-11T18:36:47.785Z %{host} 2017-05-11T18:36:47.778Z %{host} 2017-05-11T18:36:47.768Z %{host} 2017-05-11T18:36:47.757Z %{host} 2017-05-11T18:36:47.741Z %{host} 2017-05-11T18:36:47.722Z %{host} 2017-05-11T18:36:47.702Z %{host} 2017-05-11T18:36:47.690Z %{host} 2017-05-11T18:36:47.674Z %{host} 2017-05-11T18:36:47.667Z %{host} 2017-05-11T18:36:47.659Z %{host} 2017-05-11T18:36:47.649Z %{host} 2017-05-11T18:36:47.639Z %{host} 2017-05-11T18:36:47.632Z %{host} 2017-05-11T18:36:47.626Z %{host} 2017-05-11T18:36:47.619Z %{host} 2017-05-11T18:36:47.612Z %{host} 2017-05-11T18:36:47.603Z %{host} 2017-05-11T18:36:47.592Z %{host} 2017-05-11T18:36:47.555Z %{host} 2017-05-11T18:36:47.537Z %{host} 2017-05-11T18:36:47.523Z %{host} 2017-05-11T18:36:47.513Z %{host} 2017-05-11T18:36:47.505Z %{host} 2017-05-11T18:36:47.492Z %{host} 2017-05-11T18:36:47.484Z %{host} 2017-05-11T18:36:47.476Z %{host} 2017-05-11T18:36:47.468Z %{host} 2017-05-11T18:36:47.456Z %{host} 2017-05-11T18:36:47.447Z %{host} 2017-05-11T18:36:47.437Z %{host} 2017-05-11T18:36:47.425Z %{host} 2017-05-11T18:36:47.416Z %{host} 2017-05-11T18:36:47.403Z %{host} 2017-05-11T18:36:47.395Z %{host} 2017-05-11T18:36:47.388Z %{host} 2017-05-11T18:36:47.379Z %{host} 2017-05-11T18:36:47.372Z %{host} 2017-05-11T18:36:47.364Z %{host} 2017-05-11T18:36:47.351Z %{host} 2017-05-11T18:36:47.318Z %{host} 2017-05-11T18:36:47.308Z %{host} 2017-05-11T18:36:47.293Z %{host} 2017-05-11T18:36:47.274Z %{host} 2017-05-11T18:36:47.255Z %{host} 2017-05-11T18:36:47.244Z %{host} 2017-05-11T18:36:47.232Z %{host} 2017-05-11T18:36:47.215Z %{host} 2017-05-11T18:36:47.205Z %{host} 2017-05-11T18:36:47.197Z %{host} 2017-05-11T18:36:47.188Z %{host} 2017-05-11T18:36:47.180Z %{host} 2017-05-11T18:36:47.170Z %{host} 2017-05-11T18:36:47.162Z %{host} 2017-05-11T18:36:47.155Z %{host} 2017-05-11T18:36:47.147Z %{host} 2017-05-11T18:36:47.141Z %{host} 2017-05-11T18:36:47.134Z %{host} 2017-05-11T18:36:47.127Z %{host} 2017-05-11T18:36:47.119Z %{host} 2017-05-11T18:36:47.111Z %{host} 2017-05-11T18:36:47.096Z %{host} 2017-05-11T18:36:47.082Z %{host} 2017-05-11T18:36:47.064Z %{host} 2017-05-11T18:36:47.047Z %{host} 2017-05-11T18:36:47.034Z %{host} 2017-05-11T18:36:47.026Z %{host} 2017-05-11T18:36:47.014Z %{host} 2017-05-11T18:36:47.006Z %{host} 2017-05-11T18:36:46.996Z %{host} 2017-05-11T18:36:46.975Z %{host} 2017-05-11T18:36:46.965Z %{host} 2017-05-11T18:36:46.935Z %{host} 2017-05-11T18:36:46.928Z %{host} 2017-05-11T18:36:46.921Z %{host} 2017-05-11T18:36:46.914Z %{host} 2017-05-11T18:36:46.907Z %{host} 2017-05-11T18:36:46.899Z %{host} 2017-05-11T18:36:46.888Z %{host} 2017-05-11T18:36:46.867Z %{host} 2017-05-11T18:36:46.845Z %{host} 2017-05-11T18:36:46.833Z %{host} 2017-05-11T18:36:46.817Z %{host} 2017-05-11T18:36:46.798Z %{host} 2017-05-11T18:36:46.783Z %{host} 2017-05-11T18:36:46.759Z %{host} 2017-05-11T18:36:46.743Z %{host} 2017-05-11T18:36:46.725Z %{host} 2017-05-11T18:36:46.712Z %{host} 2017-05-11T18:36:46.695Z %{host} 2017-05-11T18:36:46.685Z %{host} 2017-05-11T18:36:46.677Z %{host} 2017-05-11T18:36:46.666Z %{host} 2017-05-11T18:36:46.659Z %{host} 2017-05-11T18:36:46.642Z %{host} 2017-05-11T18:36:46.631Z %{host} 2017-05-11T18:36:46.620Z %{host} 2017-05-11T18:36:46.606Z %{host} 2017-05-11T18:36:46.593Z %{host} 2017-05-11T18:36:46.577Z %{host} 2017-05-11T18:36:46.568Z %{host} 2017-05-11T18:36:46.549Z %{host} 2017-05-11T18:36:46.541Z %{host} 2017-05-11T18:36:46.532Z %{host} 2017-05-11T18:36:46.524Z %{host} 2017-05-11T18:36:46.507Z %{host} 2017-05-11T18:36:46.491Z %{host} 2017-05-11T18:36:46.474Z %{host} 2017-05-11T18:36:46.429Z %{host} 2017-05-11T18:36:46.420Z %{host} 2017-05-11T18:36:46.408Z %{host} 2017-05-11T18:36:46.397Z %{host} 2017-05-11T18:36:46.382Z %{host} 2017-05-11T18:36:46.353Z %{host} 2017-05-11T18:36:46.345Z %{host} 2017-05-11T18:36:46.337Z %{host} 2017-05-11T18:36:46.330Z %{host} 2017-05-11T18:36:46.323Z %{host} 2017-05-11T18:36:46.314Z %{host} 2017-05-11T18:36:46.302Z %{host} 2017-05-11T18:36:46.293Z %{host} 2017-05-11T18:36:46.279Z %{host} 2017-05-11T18:36:46.265Z %{host} 2017-05-11T18:36:46.252Z %{host} 2017-05-11T18:36:46.181Z %{host} 2017-05-11T18:36:46.152Z %{host} 2017-05-11T18:36:46.128Z %{host} 2017-05-11T18:36:46.110Z %{host} 2017-05-11T18:36:46.088Z %{host} 2017-05-11T18:36:46.075Z %{host} 2017-05-11T18:36:46.067Z %{host} 2017-05-11T18:36:46.058Z %{host} 2017-05-11T18:36:46.045Z %{host} 2017-05-11T18:36:46.025Z %{host} 2017-05-11T18:36:46.016Z %{host} 2017-05-11T18:36:46.008Z %{host} 2017-05-11T18:36:45.995Z %{host} 2017-05-11T18:36:45.986Z %{host} 2017-05-11T18:36:45.972Z %{host} 2017-05-11T18:36:45.934Z %{host} 2017-05-11T18:36:45.915Z %{host} 2017-05-11T18:36:45.898Z %{host} 2017-05-11T18:36:45.887Z %{host} 2017-05-11T18:36:45.855Z dc6258001c01 blah

在我终止 logstash 进程之前，这段文字会无限重复。就好像它进入了某种非终止循环。发生这种情况时，我还看到我的主题偏移量在 kafka 实例中增加。

所以这给我留下了两个问题：

1. 为什么 logstash 会一遍又一遍地重复读取同一行代码？
2. 那条消息的格式是怎么回事？不应该是timestamp host line吗？它在最后看起来是正确的，但在此之前timestamp %{host} 模式被重复了大约 232 次。为什么？

Answer 1

我自己也遇到了与 Kafka 和 RabbitMQ 类似的问题。

Logstash 不关心您将配置拆分为单独的文件，最终这样做是为了供人使用（使配置更易于查找和修改）。在运行时，所有 Logstash 配置在管道中合并在一起，并被视为单个配置。虽然我不一定知道如何应用 order，但它会在单个主机上运行：

input {
    file {
        path => ["/data/*.log"]
        start_position => "beginning"
    }
    kafka {
        topics => ["test"]
        bootstrap_servers => "<host>:9092"
        auto_offset_reset => "latest"
    }
}

output {
    kafka {
        topic_id => "test"
        bootstrap_servers => "<host>:9092"
    }
    stdout {}
}

鉴于您的其中一个输入设置为读取最后写入的偏移量，它只会读取该偏移量，不应用过滤器，并将其输出到同一主题。 Kafka 代理不知道消息本身是什么，但 Kafka（最终是 ZooKeeper）会跟踪发布者写入的消息的偏移量以及订阅者读取的消息的偏移量。

您需要运行一个单独的 Logstash 实例。要在没有第二个配置（使用标准输出）的情况​​下验证输入，我建议使用kafkacat 来读取您的 Kafka 主题（它也可以用于发布消息）。

kafkacat -C -b <broker>:9092 -t <topic>

允许您使用单个 Logstash 实例的唯一替代方法（我知道有效）是您必须在 filter 步骤中使用 mutate 方法，但这会使逻辑复杂化，最好使用 Logstash 遵守托运人/索引器模式。