一、 框架图 2

二、 ELK简介 3

三、 环境要求 3

四、 系统搭建 4

1.Java环境 4

2.安装elasticsearch 4

1)安装elasticsearch 4

4)内核调优 4

5)配置head插件 5

6修改配置文件 6

7)浏览器访问 6

3.安装logstash 7

1)yum安装服务 7

2)pipeline文件 7

4)后台验证 8

4.安装kibana 8

1)yum安装kibana 8

2)浏览器访问验证 9

 

 

 

一、框架图

 ELK日志监控

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

二、ELK简介

ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成:

ElasticSearch是一个基于Lucene的开源分布式搜索服务器。它的特点有:分布式,零配置,自动发现,索引自动分片,索引副本机制,restful风格接口,多数据源,自动搜索负载等。它提供了一个分布式多用户能力的全文搜索引擎,基于RESTful web接口。Elasticsearch是用Java开发的,并作为Apache许可条款下的开放源码发布,是第二流行的企业搜索引擎。设计用于云计算中,能够达到实时搜索,稳定,可靠,快速,安装使用方便。在elasticsearch中,所有节点的数据是均等的。

Logstash是一个完全开源的工具,他可以对你的日志进行收集、过滤、分析,并将其存储供以后使用(如,搜索),您可以使用它。说到搜索,logstash带有一个web界面,搜索和展示所有日志。

Kibana 是一个基于浏览器页面的Elasticsearch前端展示工具,也是一个开源和免费的工具,它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面,可以帮助您汇总、分析和搜索重要数据日志。

 

 

 

 

 

 

三、环境要求

CentOS 6.9 

Elasticsearch 6.1

Logstash 6.1

Kibana 6.1

Jdk 1.8.0

 

 

 

 

 

 

 

 

 

 

四、系统搭建

1.Java环境

[[email protected] ~]# yum install -y java-1.8.0

[[email protected] ~]# java -version

openjdk version "1.8.0_121"

OpenJDK Runtime Environment (build 1.8.0_121-b13)

OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)

 

2.安装elasticsearch

1)安装elasticsearch

[[email protected] ~]# tar -xf elasticsearch-6.1.0.tar.gz -C /usr/local/ && mv /usr/local/elasticsearch-6.1.0 /usr/local/elasticsearch

[[email protected] ~]# vim /usr/local/elasticsearch/config/elasticsearch.yml

cluster.name: elk     //集群名

node.name: elk-node1   //节点名

path.data: /usr/local/elasticsearch/data//data存储路径

path.logs: /var/log/elasticsearch   //log存储路径

network.host: 0.0.0.0    //监听地址

http.port: 9200          //监听端口

discovery.zen.ping.unicast.hosts: ["192.168.1.20", "192.168.1.11"]         //elasticsearch服务器集群节点地址

discovery.zen.minimum_master_nodes: 2     //2个节点安装elasticsearch服务

http.cors.enabled: true   

http.cors.allow-origin: "*"

bootstrap.memory_lock: false//Centos7以下版本配置
bootstrap.system_call_filter: false//Centos7以下版本配置

 

4)内核调优

vi /etc/security/limits.conf 

#添加如下内容:

* soft nofile 65536

* hard nofile 131072

* soft nproc 20480

* hard nproc 40960

 

vi /etc/security/limits.d/90-nproc.conf 

#修改以下内容

* soft nproc 10240

 

vi /etc/sysctl.conf 

#添加下面配置:

vm.max_map_count=655360

 

 

#配置完成后,执行以下命令:

sysctl -p

 

 

5)配置head插件

[[email protected] ~]# mkdir  /usr/local/elasticsearch/data /var/log/elasticsearch &&

wget https://nodejs.org/dist/v9.3.0/node-v9.3.0-linux-x64.tar.gz

[[email protected] ~]# tar -xf node-v9.3.0-linux-x64.tar.gz -C /usr/local/

[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/bin/node /usr/bin/node

[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/bin/npm /usr/bin/npm

[[email protected] ~]# node -v

v6.9.5

[[email protected] ~]# npm -v

3.10.10

[[email protected] ~]# npm install -g grunt-cli

[[email protected] ~]# ln -s /usr/local/node-v9.3.0-linux-x64/lib/node_modules/grunt-cli/bin/grunt /usr/bin/grunt

[[email protected] ~]# cd /usr/local/elasticsearch/

[[email protected] elasticsearch]# yum install -y git

[[email protected] elasticsearch]# git clone git://github.com/mobz/elasticsearch-head.git

[[email protected] elasticsearch]# useradd elasticsearch && chown -R elasticsearch:elasticsearch elasticsearch-head/

[[email protected] elasticsearch]# cd /usr/local/elasticsearch/elasticsearch-head/ && npm install [email protected] --ignore-scripts

[[email protected] elasticsearch-head]# npm install

[[email protected] elasticsearch-head]# cd /usr/local/elasticsearch/elasticsearch-head/_site/

[[email protected] _site]# cp app.js app.js.bak

[[email protected] _site]# vim app.js

                init: function(parent) {

                        this._super();

                        this.prefs = services.Preferences.instance();

                        this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.1.20:9200";   //修改4354

[[email protected] _site]# cd /usr/local/elasticsearch/elasticsearch-head/

[[email protected] elasticsearch-head]# cp Gruntfile.js Gruntfile.js.bak

[[email protected] elasticsearch-head]# vim Gruntfile.js

                connect: {

                        server: {

                                options: {

                                        hostname: "0.0.0.0",   //添加这一行

                                        port: 9100,

                                        base: '.',

                                        keepalive: true

                                }

                        }

                }

 

 

6)修改配置文件

修改文件:/usr/local/elasticsearch/config/elasticsearch.yml,在Memory字段下增加内容:

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

[[email protected] elasticsearch-head]# /usr/local/elasticsearch/bin/elasticsearch  &  //不能以root用户启动

[[email protected] elasticsearch-head]# grunt server & //需要在elasticsearch-head目录下启动该服务

[[email protected] elasticsearch-head]# netstat -anplut  //检查9200,9300,9100端口

 

 

 

7)浏览器访问

访问http://192.168.1.20:9200/进行验证 

 ELK日志监控

访问http://192.168.1.20:9100/进行验证 

 ELK日志监控

3.安装logstash

1)yum安装服务

[[email protected] ~]# tar xf logstash-6.1.0.tar.gz -C /usr/local/ && mv /usr/local/logstash/logstash-6.1.0 /usr/local/logstash/logstash

[[email protected] ~]# mkdir /var/log/logstash && cd /usr/local/logstash/

[[email protected] logstash]# vim /usr/local/logstash/config/logstash.yml

path.data: /usr/local/logstash/data

path.config: /usr/local/logstash/config

path.logs: /var/log/logstash

 

2)pipeline文件

[[email protected] logstash]# cd /usr/local/logstash/config/

[[email protected] conf.d]# vim messages.conf

input {

    file {

        path => "/var/log/messages"

    }

}

output {

    elasticsearch {

        hosts => ["192.168.1.20:9200"]

        index => "messages-%{+YYYY.MM.dd}"

    }

    stdout {# codec => rubydebug

    }

}

[[email protected] logstash]# chown -R logstash:logstash /usr/local/logstash/config/

[[email protected] logstash]# chmod 644 /var/log/messages

 

 

 

 

 

[[email protected] logstash]# useradd logstash && chown -R logstash:root /usr/local/logstash/config/

[[email protected] logstash]# /usr/local/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

 

 

 

 

 

启动logstash并验证

[[email protected] ~]# /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/messages.conf &

Created symlink from /etc/systemd/system/multi-user.target.wants/logstash.service to /etc/systemd/system/logstash.service.

[[email protected] ~]# netstat -anplut

//查看9600端口

 

 

 

 

 

 

 

4)后台验证

访问网址 http://192.168.1.20:9100/ 

 ELK日志监控

 

4.安装kibana

1)yum安装kibana

[[email protected] ~]# tar xf kibana-6.1.0-linux-x86_64.tar.gz -C /usr/local/ && mv /usr/local/kibana-6.1.0 /usr/local/kibana

[[email protected] ~]# cd /usr/local/kibana

[[email protected] kibana]# vim /usr/local/kibana/config/kibana.yml

server.port: 5601

server.host: "0.0.0.0"

elasticsearch.url: "http://192.168.1.20:9200"

[[email protected] kibana]# /usr/local/kibana/bin/kibana &

[[email protected] kibana]# netstat -anplut |grep :5601

tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      4335/node

2)浏览器访问验证

访问网址 http://192.168.1.20:5601/ 

ELK日志监控

【注意:在红线部分添加messages-*,然后点击Discover,如果不能显示图形,请把时间选择一小时或更长。】

ELK日志监控 

 

分类:

技术点:

相关文章: