Logstash 不读取文件输入的整行

Question

我正在使用 Logstash，但在让一个相当简单的配置工作时遇到了麻烦。

input {
  file {
    path => "C:/path/test-data/*.log"
    start_position => beginning
    type => "usage_data"
  }
}

filter {
  if [type] == "usage_data" {
    grok {
      match => { "message" => "^\s*%{NUMBER:lineNumber}\s+%{TIMESTAMP_ISO8601:date},(?<value1>[A-Za-z0-9+/]+),(?<value2>[A-Za-z0-9+/]+),(?<value3>[A-Za-z0-9+/]+),(?<value4>[^,]+),(?<value5>[^\r]*)" }
    }
  }

  if "_grokparsefailure" not in [tags] {
    drop { }
  }
}

output {
  stdout { codec => rubydebug }
}

我这样称呼 Logstash：

SET LS_MAX_MEM=2g

DEL "%USERPROFILE%\.sincedb_*" 2> NUL
"C:\Program Files (x86)\logstash-1.4.1\bin\logstash.bat" agent -p "C:\path\\." -w 1 -f "logstash.conf"

输出：

←[33mUsing milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:w
arn}←[0m
{
       "message" => ",",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:16:08.591Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20141116223000.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

如果我只解析C:\path\test-data\monitor_20141116223000.log，所有行都被读取并且没有grokparsefailure。如果我删除 C:\path\test-data\monitor_20141116223000.log 相同的 grokparsefailure 会在另一个日志文件中弹出：

{
       "message" => "atches in another context\r",
      "@version" => "1",
    "@timestamp" => "2014-11-20T09:14:04.779Z",
          "type" => "usage_data",
          "host" => "my-machine",
          "path" => "C:/path/test-data/monitor_20140829235900.log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

特别是最后一个输出证明 Logstash 没有读取整行或尝试解释没有的换行符。它总是在同一位置的同一行中断。

也许我应该补充一点，日志文件包含 \n 作为行分隔符，并且我在 Windows 上运行 Logstash。但是，我没有收到很多错误，只是那个错误。而且里面有很多台词。当我删除 if "_grokparsefailure" ... 时，它们都会正确显示。

我认为缓冲存在一些问题，但我不知道如何进行这项工作。有什么想法吗？

Answer 1

解决方法：

# diff -Nur /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb
--- /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb.orig       2015-02-25 10:46:06.916321816 +0700
+++ /opt/logstash/vendor/bundle/jruby/1.9/gems/filewatch-0.5.1/lib/filewatch/tail.rb    2015-02-12 18:39:34.943833909 +0700
@@ -86,7 +86,9 @@
           _read_file(path, &block)
           @files[path].close
           @files.delete(path)
-          @statcache.delete(path)
+          #@statcache.delete(path)
+          inode = @statcache.delete(path)
+          @sincedb[inode] = 0
         else
           @logger.warn("unknown event type #{event} for #{path}")
         end