如何通过 curl 查询 Logstash 并仅返回特定字段

Question

现在我正在使用“match_all”查询来获取 Logstash 正在处理的数据。我得到的输出是作为事件一部分的每个字段，因为它应该是。这是我的查询：

{
"query": {
    "match_all" : { }
},
  "size": 1,
  "sort": [
{
 "@timestamp": {
     "order": "desc"
  }
  }
  ]
}

如您所见，我也在对我的结果进行排序，我总是得到最近输出的结果。

这是我的输出示例：

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 15768,
    "max_score" : null,
    "hits" : [
      {
        "_index" : "filebeat-2017.02.24",
        "_type" : "bro",
        "_id" : "AVpx-pFtiEtl3Zqhg8tF",
        "_score" : null,
        "_source" : {
          "resp_pkts" : 0,
          "source" : "/usr/local/bro/logs/current/conn.log",
          "type" : "bro",
          "id_orig_p" : 56058,
          "duration" : 848.388112,
          "local_resp" : true,
          "uid" : "CPndOf4NNf9CzTILFi",
          "id_orig_h" : "192.168.137.130",
          "conn_state" : "OTH",
          "@version" : "1",
          "beat" : {
            "hostname" : "localhost.localdomain",
            "name" : "localhost.localdomain",
            "version" : "5.2.0"
          },
          "host" : "localhost.localdomain",
          "id_resp_h" : "192.168.137.141",
          "id_resp_p" : 22,
          "resp_ip_bytes" : 0,
          "offset" : 115612,
          "orig_bytes" : 32052,
          "local_orig" : true,
          "input_type" : "log",
          "orig_ip_bytes" : 102980,
          "orig_pkts" : 1364,
          "missed_bytes" : 0,
          "history" : "DcA",
          "tunnel_parents" : [ ],
          "message" : "{\"ts\":1487969779.653504,\"uid\":\"CPndOf4NNf9CzTILFi\",\"id_orig_h\":\"192.168.137.130\",\"id_orig_p\":56058,\"id_resp_h\":\"192.168.137.141\",\"id_resp_p\":22,\"proto\":\"tcp\",\"duration\":848.388112,\"orig_bytes\":32052,\"resp_bytes\":0,\"conn_state\":\"OTH\",\"local_orig\":true,\"local_resp\":true,\"missed_bytes\":0,\"history\":\"DcA\",\"orig_pkts\":1364,\"orig_ip_bytes\":102980,\"resp_pkts\":0,\"resp_ip_bytes\":0,\"tunnel_parents\":[]}",
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "@timestamp" : "2017-02-24T21:15:29.414Z",
          "resp_bytes" : 0,
          "proto" : "tcp",
          "fields" : {
            "sensorType" : "networksensor"
          },
          "ts" : 1.487969779653504E9
        },
        "sort" : [
          1487970929414
        ]
      }
    ]
  }
}

如您所见，这是要在外部应用程序中处理的 很多 输出（用 C# 编写，因此所有这些字符串上的垃圾收集都很大），我就是不这样做需要。

我的问题是，如何设置我的查询，以便只获取我需要的字段？

Answer 1

对于 5.x，有一个更改允许您进行 _source 过滤。其文档是 here，它看起来像这样：

{ 
 "query": {
   "match_all" : { }
 },
 "size": 1,
 "_source": ["a","b"],
 ...

结果如下：

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "xxx",
        "_type" : "xxx",
        "_id" : "xxx",
        "_score" : 1.0,
        "_source" : {
          "a" : 1,
          "b" : "2"
        }
      }
    ]
  }
}

对于 5 之前的版本，您可以使用 fields 参数：

您的查询可以在查询的根级别传递,"fields": ["field1","field2"...]。它返回的格式会有所不同，但它会起作用。

{ 
"query": {
  "match_all" : { }
},
"size": 1,
"fields": ["a","b"],
...

这将产生如下输出：

  {
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "hits": {
    "total": 2077,
    "max_score": 1,
    "hits": [
      {
        "_index": "xxx",
        "_type": "xxx",
        "_id": "xxxx",
        "_score": 1,
        "fields": {
          "a": [
            0
          ],
          "b": [
            "xyz"
          ]
        }
      }
    ]
  }
}

字段始终是数组（自 1.0 API 起）并且没有任何方法可以改变它，因为 Elasticsearch 天生就具有多值感知能力。