一、框架图
二、ELK简介
ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成:
ElasticSearch是一个基于Lucene的开源分布式搜索服务器。它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口。Elasticsearch是用Java开发的,并作为Apache许可条款下的开放源码发布,是第二流行的企业搜索引擎。设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便。在elasticsearch中,所有节点的数据是均等的。
Logstash是一个完全开源的工具,他可以对你的日志进行收集、过滤、分析,并将其存储供以后使用(如,搜索),您可以使用它。说到搜索,logstash带有一个web界面,搜索和展示所有日志。
Kibana 是一个基于浏览器页面的Elasticsearch前端展示工具,也是一个开源和免费的工具,它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助您汇总、分析和搜索重要数据日志。
三、环境要求
CentOS 6.9
Elasticsearch 6.1
Logstash 6.1
Kibana 6.1
Jdk 1.8.0
四、系统搭建
1.Java环境
[[email protected] ~]# yum install -y java-1.8.0
[[email protected] ~]# java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b13)
OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
2.安装elasticsearch
(1)安装elasticsearch
[[email protected] ~]# tar -xf elasticsearch-6.1.0.tar.gz -C /usr/local/ && mv /usr/local/elasticsearch-6.1.0 /usr/local/elasticsearch
[[email protected] ~]# vim /usr/local/elasticsearch/config/elasticsearch.yml
cluster.name: elk //集群名
node.name: elk-node1 //节点名
path.data: /usr/local/elasticsearch/data//data存储路径
path.logs: /var/log/elasticsearch //log存储路径
network.host: 0.0.0.0 //监听地址
http.port: 9200 //监听端口
discovery.zen.ping.unicast.hosts: ["192.168.1.20", "192.168.1.11"] //elasticsearch服务器集群节点地址
discovery.zen.minimum_master_nodes: 2 //2个节点安装elasticsearch服务
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: false//Centos7以下版本配置
bootstrap.system_call_filter: false//Centos7以下版本配置
(4)内核调优
vi /etc/security/limits.conf
#添加如下内容:
* soft nofile 65536
* hard nofile 131072
* soft nproc 20480
* hard nproc 40960
vi /etc/security/limits.d/90-nproc.conf
#修改以下内容
* soft nproc 10240
vi /etc/sysctl.conf
#添加下面配置:
vm.max_map_count=655360
#配置完成后,执行以下命令:
sysctl -p
(5)配置head插件
[[email protected] ~]# mkdir /usr/local/elasticsearch/data /var/log/elasticsearch &&
wget https://nodejs.org/dist/v9.3.0/node-v9.3.0-linux-x64.tar.gz
[[email protected] ~]# tar -xf node-v9.3.0-linux-x64.tar.gz -C /usr/local/
[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/bin/node /usr/bin/node
[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/bin/npm /usr/bin/npm
[[email protected] ~]# node -v
v6.9.5
[[email protected] ~]# npm -v
3.10.10
[[email protected] ~]# npm install -g grunt-cli
[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/lib/node_modules/grunt-cli/bin/grunt /usr/bin/grunt
[[email protected] ~]# cd /usr/local/elasticsearch/
[[email protected] elasticsearch]# yum install -y git
[[email protected] elasticsearch]# git clone git://github.com/mobz/elasticsearch-head.git
[[email protected] elasticsearch]# useradd elasticsearch && chown -R elasticsearch:elasticsearch elasticsearch-head/
[[email protected] elasticsearch]# cd /usr/local/elasticsearch/elasticsearch-head/ && npm install [email protected] --ignore-scripts
[[email protected] elasticsearch-head]# npm install
[[email protected] elasticsearch-head]# cd /usr/local/elasticsearch/elasticsearch-head/_site/
[[email protected] _site]# cp app.js app.js.bak
[[email protected] _site]# vim app.js
init: function(parent) {
this._super();
this.prefs = services.Preferences.instance();
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.1.20:9200"; //修改4354行
[[email protected] _site]# cd /usr/local/elasticsearch/elasticsearch-head/
[[email protected] elasticsearch-head]# cp Gruntfile.js Gruntfile.js.bak
[[email protected] elasticsearch-head]# vim Gruntfile.js
connect: {
server: {
options: {
hostname: "0.0.0.0", //添加这一行
port: 9100,
base: '.',
keepalive: true
}
}
(6)修改配置文件
修改文件:/usr/local/elasticsearch/config/elasticsearch.yml,在Memory字段下增加内容:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
[[email protected] elasticsearch-head]# /usr/local/elasticsearch/bin/elasticsearch & //不能以root用户启动
[[email protected] elasticsearch-head]# grunt server & //需要在elasticsearch-head目录下启动该服务
[[email protected] elasticsearch-head]# netstat -anplut //检查9200,9300,9100端口
(7)浏览器访问
访问http://192.168.1.20:9200/进行验证
访问http://192.168.1.20:9100/进行验证
3.安装logstash
(1)yum安装服务
[[email protected] ~]# tar xf logstash-6.1.0.tar.gz -C /usr/local/ && mv /usr/local/logstash/logstash-6.1.0 /usr/local/logstash/logstash
[[email protected] ~]# mkdir /var/log/logstash && cd /usr/local/logstash/
[[email protected] logstash]# vim /usr/local/logstash/config/logstash.yml
path.data: /usr/local/logstash/data
path.config: /usr/local/logstash/config
path.logs: /var/log/logstash
(2)pipeline文件
[[email protected] logstash]# cd /usr/local/logstash/config/
[[email protected] conf.d]# vim messages.conf
input {
file {
path => "/var/log/messages"
}
}
output {
elasticsearch {
hosts => ["192.168.1.20:9200"]
index => "messages-%{+YYYY.MM.dd}"
}
stdout {# codec => rubydebug
}
}
[[email protected] logstash]# chown -R logstash:logstash /usr/local/logstash/config/
[[email protected] logstash]# chmod 644 /var/log/messages
[[email protected] logstash]# useradd logstash && chown -R logstash:root /usr/local/logstash/config/
[[email protected] logstash]# /usr/local/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
启动logstash并验证
[[email protected] ~]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/messages.conf &
Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.
[[email protected] ~]# netstat -anplut
//查看9600端口
(4)后台验证
访问网址 http://192.168.1.20:9100/
4.安装kibana
(1)yum安装kibana
[[email protected] ~]# tar xf kibana-6.1.0-linux-x86_64.tar.gz -C /usr/local/ && mv /usr/local/kibana-6.1.0 /usr/local/kibana
[[email protected] ~]# cd /usr/local/kibana
[[email protected] kibana]# vim /usr/local/kibana/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.1.20:9200"
[[email protected] kibana]# /usr/local/kibana/bin/kibana &
[[email protected] kibana]# netstat -anplut |grep :5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 4335/node
(2)浏览器访问验证
访问网址 http://192.168.1.20:5601/
【注意:在红线部分添加messages-*,然后点击Discover,如果不能显示图形,请把时间选择一小时或更长。】