【发布时间】：2016-04-05 08:47:20
【问题描述】：
我无法通过结合纬度和经度在 Logstash 中构成 geo_point。我遵循了其他人的指示，但看起来这些示例是基于旧版本的 ELK。自 ELK 2.2 以来，geo_point 发生了重大变化，我不确定我是否以正确的方式执行了所有步骤。下面我解释一下我的设置。
我使用的ELK版本是：
curl -XGET 'localhost:9200'
{
"name" : "Artie",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.2.1",
"build_hash" : "d045fc29d1932bce18b2e65ab8b297fbf6cd41a1",
"build_timestamp" : "2016-03-09T09:38:54Z",
"build_snapshot" : false,
"lucene_version" : "5.4.1"
},
"tagline" : "You Know, for Search"
}
在 Docker 容器上使用 Elasticsearch、Logstash 和 Kibana，但这不重要。
这就是我的 logstash.conf 的样子：
cat logstash.conf
input {
http_poller {
urls => {
myresource => "myhost/data.json"
}
request_timeout => 1
interval => 1
# Parse every line captured from data.json as a new event.
codec => "line"
}
}
filter {
if [message] !~ /\"hex\":/ {
# drop messages without "hex"
drop {}
}
# Capture "hex":72d5a1
grok {
match => { "message" => "\"hex\":\"(?<hex>[^\"]+)\"," }
}
mutate {
convert => { "hex" => "string" }
}
# Capture "lat":50.047613
if [message] =~ /\"lat\":/ {
grok {
match => { "message" => "\"lat\":(?<latitude>[^,]+),"}
}
mutate {
convert => { "latitude" => "float" }
}
}
# Capture "lon":1.702955
if [message] =~ /\"lon\":/ {
grok {
match => { "message" => "\"lon\":(?<longitude>[^,]+)," }
}
mutate {
convert => { "longitude" => "float" }
}
}
# convert latitude and longitude into location.
mutate {
rename => {
"longitude" => "[location][lon]"
"latitude" => "[location][lat]"
}
}
mutate {
remove_field => [ "message" ]
}
}
output {
elasticsearch {
hosts => [ "elasticsearchhost:9200" ]
index => "logstash-%{+YYYY.MM.dd}"
}
}
重要的部分是“lon”和“lat”是从“消息”中捕获的，并且它们被格式化为“位置”字段。
当我查询 elasticsearch 时，我得到了这种记录：
{
"_index": "logstash-2016.04.04",
"_type": "logs",
"_id": "AVPieJtgVkabtr-H2szZ",
"_score": null,
"_source": {
"@version": "1",
"@timestamp": "2016-04-04T18:11:07.857Z",
"hex": "3e37aa",
"location": {
"lon": 4.8246,
"lat": 52.329208
}
},
"fields": {
"@timestamp": [
1459793467857
]
},
"sort": [
1459793467857
]
}
从我在文档中读到的符号"location": { "lon": 4.8246, "lat": 52.329208 } 看起来不错。但问题是我无法在 Kibana 中选择“位置”字段作为 geo_point。
根据 ELK 文档，我需要确保“位置”字段映射到 geo_point 类型。它要求启用 doc_values 才能正常工作。我不确定我是否应该做点什么，因为当我查看模板时，默认情况下“位置”字段似乎已经映射："location" : { "type" : "geo_point", "doc_values" : true }
这是我的模板的样子：
# curl -XGET localhost:9200/_template/logstash?pretty
{
"logstash" : {
"order" : 0,
"template" : "logstash-*",
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"message_field" : {
"mapping" : {
"fielddata" : {
"format" : "disabled"
},
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
},
"match_mapping_type" : "string",
"match" : "message"
}
}, {
"string_fields" : {
"mapping" : {
"fielddata" : {
"format" : "disabled"
},
"index" : "analyzed",
"omit_norms" : true,
"type" : "string",
"fields" : {
"raw" : {
"ignore_above" : 256,
"index" : "not_analyzed",
"type" : "string",
"doc_values" : true
}
}
},
"match_mapping_type" : "string",
"match" : "*"
}
}, {
"float_fields" : {
"mapping" : {
"type" : "float",
"doc_values" : true
},
"match_mapping_type" : "float",
"match" : "*"
}
}, {
"double_fields" : {
"mapping" : {
"type" : "double",
"doc_values" : true
},
"match_mapping_type" : "double",
"match" : "*"
}
}, {
"byte_fields" : {
"mapping" : {
"type" : "byte",
"doc_values" : true
},
"match_mapping_type" : "byte",
"match" : "*"
}
}, {
"short_fields" : {
"mapping" : {
"type" : "short",
"doc_values" : true
},
"match_mapping_type" : "short",
"match" : "*"
}
}, {
"integer_fields" : {
"mapping" : {
"type" : "integer",
"doc_values" : true
},
"match_mapping_type" : "integer",
"match" : "*"
}
}, {
"long_fields" : {
"mapping" : {
"type" : "long",
"doc_values" : true
},
"match_mapping_type" : "long",
"match" : "*"
}
}, {
"date_fields" : {
"mapping" : {
"type" : "date",
"doc_values" : true
},
"match_mapping_type" : "date",
"match" : "*"
}
}, {
"geo_point_fields" : {
"mapping" : {
"type" : "geo_point",
"doc_values" : true
},
"match_mapping_type" : "geo_point",
"match" : "*"
}
} ],
"_all" : {
"omit_norms" : true,
"enabled" : true
},
"properties" : {
"@timestamp" : {
"type" : "date",
"doc_values" : true
},
"geoip" : {
"dynamic" : true,
"type" : "object",
"properties" : {
"ip" : {
"type" : "ip",
"doc_values" : true
},
"latitude" : {
"type" : "float",
"doc_values" : true
},
"location" : {
"type" : "geo_point",
"doc_values" : true
},
"longitude" : {
"type" : "float",
"doc_values" : true
}
}
},
"@version" : {
"index" : "not_analyzed",
"type" : "string",
"doc_values" : true
}
}
}
},
"aliases" : { }
}
}
我没有在此模板中添加任何内容。这是全新安装 Logstash 和 Elastic 并使用我的 logstash.conf 文件启动 Logstash 后的样子。
我的问题是：我需要采取哪些步骤来解决我的问题？
非常感谢！
【问题讨论】：
标签： json logstash logstash-configuration