在输入日志文件中解析日期答案

【问题标题】：Parse date inside input logs file在输入日志文件中解析日期
【发布时间】：2019-09-05 16:50:22
【问题描述】：

我是使用 ELK 堆栈的新手，我正在尝试从 S3 文件创建索引。此 S3 文件的格式为 CSV，并具有以下架构：

date: Date field with format yyyy-MM-dd HH:mm:ss
filename: Name of the input file that triggers some events
input_registers: count with num of lines for the file
wrong_registers: count with num of wrong registers
result_registers: count with num of result registers (validated)

我需要将date 设置为ELK 上的@timestamp 字段。

我已经用日期过滤插件尝试了一些东西，这里我展示了我当前的配置：

input{
    s3 {
        "id" => "rim-pfinal"
        "access_key_id" => ""
        "secret_access_key" => ""
        "region" => "eu-west-3"
        "bucket" => "practica.final.rim.elk"
        "prefix" => "logs"
        "interval" => "3600"
        "additional_settings" => {
            "force_path_style" => true
            "follow_redirects" => false
        }
        sincedb_path => "/dev/null"
    }
}

filter {
  date {
      match => [ "date", "ISO8601", "yyyy-MM-dd HH:mm:ss" ]
      target => "date"
      add_field => { "DummyField" => "Fecha cambiada" }
  }
  csv{
    columns => ["date", "filename", "input_registers", "wrong_registers", "result_registers", "err_type"]
    separator => ";"
  }
  mutate { convert => [ "input_registers", "integer"] }
  mutate { convert => [ "wrong_registers", "integer"] }
  mutate { convert => [ "result_registers", "integer"] }

  #Remove first header line to insert in elasticsearch
  if [PK] =~ "PK"{
    drop {}
  }
}

output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "practica-rim"
    }
}

我尝试将 target 设置为时间戳和匹配，但似乎不起作用。

感谢您的帮助！

【问题讨论】：

你的弹性搜索版本是多少？
将日期过滤器放在 csv 过滤器之后。

标签： csv date elasticsearch logstash elastic-stack

【解决方案1】：

{
   "query":{
      "range":{
         "@timestamp":{
            "gte":"2015-08-04T11:00:00",
            "lt":"2015-08-04T12:00:00"
         }
      }
   }
}

datetimes will be serialized
es.index(index="my-index", doc_type="test-type", id=42, body={"any": "data", "timestamp": datetime.now()})
{u'_id': u'42', u'_index': u'my-index', u'_type': u'test-type', u'_version': 1, u'ok': True}

# but not deserialized
>>> es.get(index="my-index", doc_type="test-type", id=42)['_source']
{u'any': u'data', u'timestamp': u'2013-05-12T19:45:31.804229'}

https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

【讨论】：