ELK 堆栈没有保留一些文档答案

【问题标题】：ELK stack not persisting some documentsELK 堆栈没有保留一些文档
【发布时间】：2015-02-03 05:48:05
【问题描述】：

我有 logstash 将事件推送到 elasticsearch，有些工作，有些没有。

这是一个确实出现在 elasticsearch 中的文档示例：

{
       "Application" => "API",
       "Environment" => "myenv",
        "LoggerName" => "MyLoggerName",
        "@timestamp" => "2015-02-01T17:18:30.454Z",
          "LogLevel" => "Warn",
      "DeploymentId" => "ebb9e128b8d44994b7bbbf27b6893b03",
    "RoleInstanceId" => "MyRoleInstance",
           "Message" => "The message.",
          "@version" => "1"
}

但是以下一个（以及更多类似的）没有：

    {
           "Application" => "API",
           "Environment" => "myenv",
            "LoggerName" => "Common.Services.RequestLogger",
            "@timestamp" => "2015-02-01T17:19:46.265Z",
              "LogLevel" => "Info",
          "DeploymentId" => "0a56017c4ad14cfe818afdbc52dabe76",
        "RoleInstanceId" => "Instance",
                  "Data" => {
                       "Elapsed" => "PT0.0119377S",
                    "RequestDto" => {
                "__type" => "Structure.Definition.LoginRequest, Structure.Definition",
                 "Email" => "email@domain.com"
            },
                   "ResponseDto" => {
                               "__type" => "ServiceStack.HttpResult, ServiceStack",
                              "Headers" => {},
                "AllowsPartialResponse" => false,
                              "Options" => {},
                               "Status" => 200,
                           "StatusCode" => "OK",
                             "Response" => {
                            "__type" => "Structure.Definition.LoginResponse, Structure.Definition",
                            "UserId" => "xxxx",
                    "OrganisationId" => "xxxx",
                             "Token" => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                            "TimeMs" => 0
                },
                       "ResponseFilter" => {


                  "__type" => "ServiceStack.Host.ContentTypes, Service
Stack",
                "ContentTypeFormats" => {
                           "csv" => "text/csv",
                      "markdown" => "text/markdown",
                         "plain" => "text/plain",
                    "x-protobuf" => "application/x-protobuf"
                }
            },
                    "PaddingLength" => 0,
                     "IsPartialRequest" => false
            },
                 "OperationName" => "LoginRequest",
                          "Verb" => "POST",
                   "AbsoluteUri" => "https://xxxx/auth/login",
                        "RawUrl" => "/auth/login",
                       "IsLocal" => false,
            "IsSecureConnection" => true,
                      "RemoteIp" => "000.000.000.000",
               "UserHostAddress" => "000.000.000.000",
                     "UserAgent" => "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit
    /537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36",
                    "StatusCode" => 200,
             "StatusDescription" => "OK",
                          "Type" => "Request"
        },
              "@version" => "1"
    }

这些是从 stdout { codec => rubydebug {} } 复制的，所以我知道 logstash 理解该文档，但是第二个类型从未出现在 elasticsearch 中。

我查看了这是否与文件大小有关，但我没有发现任何可以限制这一点的东西。

这里是logstash配置：

input {
  redis {
    codec => json {}
    data_type => "list"
    host => "xxxx.redis.cache.windows.net"
    key => "logstash"
    password => "xxxxxxx"
  }
  redis {
    codec => json {}
    data_type => "list"
    host => "xxxx.redis.cache.windows.net"
    key => "logstash"
    password => "xxxxxx"
  }
}
output {
  elasticsearch {
    host => "127.0.0.1"
    port => "9201"
    protocol => http
  }
  stdout { codec => rubydebug{} }
}

我在 elasticsearch 配置中缺少什么吗？ logstash 节点通过 SSH 端口映射通过 HTTP 连接到 elasticsearch。

【问题讨论】：

能否添加从 Elasticsearch 获取的这种索引类型的映射。
这里@VineethMohan 是映射类型的要点，看起来它包含了曾经发布过的所有字段：gist.github.com/jamescarter-le/17dfa86410b3cc0d2687

标签： elasticsearch logstash

【解决方案1】：

在失败的文档中，您似乎有名为“__type”的字段。以下划线开头的字段受到限制，并且会导致问题。因此，我建议您将“__type”字段重命名为“type”，然后再试一次。此处描述了此字段名称限制和其他字段名称限制：http://grokbase.com/t/gg/elasticsearch/144s6e3877/illegal-characters-in-elasticsearch-field-names

【讨论】：

我发现如果我在本地启动一个新的 elasticsearch 实例，这些消息会被记录下来，所以我认为这与服务器配置或它如何解释消息有关？