ELK日志分析系统

一、安装Logstash

1.安装jdk

Logstash的运行依赖于Java环境

[[email protected] ~]# yum -y install java-1.8.0

[[email protected] ~]# java -version

openjdk version "1.8.0_131"

OpenJDK Runtime Environment (build 1.8.0_131-b11)

OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

2.安装Logstash

[[email protected] ~]# cd /opt/software/

[[email protected] software]# wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz

[[email protected] software]# tar zxvf logstash-1.5.4.tar.gz -C /usr/local/

3.配置Logstash的环境变量

[[email protected] software]# echo "export PATH=\$PATH:/usr/local/logstash-1.5.4/bin" > /etc/profile.d/logstash.sh

[[email protected] software]# . /etc/profile

[[email protected] software]#

4.启动Logstash(注:-e:指定Logstash的配置信息,可以用于快速测试;-f:指定Logstash的配置文件,可以用于生产环境。)

4.1 -e参数指定logstash的配置信息,用于快速测试,直接输出到屏幕

[[email protected] software]# logstash -e "input {stdin{}} output {stdout{}}"

 

4.2通过-e参数指定logstash的配置信息,用于快速测试,以json格式输出到屏幕

[[email protected] software]# logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

 ELK日志分析

5.logstash以配置文件方式启动

5.1输出信息到屏幕

[[email protected] software]# vim logstash-simple.conf

ELK日志分析

[[email protected] software]# cat logstash-simple.conf

ELK日志分析

input { stdin {} }

output {

   stdout { codec=> rubydebug }

}

普通方式启动

[[email protected] software]# logstash -f logstash-simple.conf

 

debug模式开启

[[email protected] software]# logstash agent -f logstash-simple.conf --verbose

 

5.2 logstash输出信息存储到redis数据库中

[[email protected] software]# vim logstash_to_redis.conf

[[email protected] software]# cat logstash_to_redis.conf

input { stdin { } }

output {

    stdout { codec => rubydebug }

    redis {

        host => '192.168.11.43'

        data_type => 'list'

        key => 'logstash:redis'

    }

}

二、Redis

1.安装Redis

[[email protected] software]# wget http://download.redis.io/releases/redis-2.8.19.tar.gz

[[email protected] software]# yum -y install tcl

[[email protected] software]# tar zxvf redis-2.8.19.tar.gz

[[email protected] software]# cd redis-2.8.19

[[email protected] redis-2.8.19]# make MALLOC=libc

 

安装gcc

[[email protected] software]# yum -y install gcc

[[email protected] software]# cd redis-2.8.19

[[email protected] redis-2.8.19]# make MALLOC=libc

 

[[email protected] redis-2.8.19]# make test

[[email protected] redis-2.8.19]# make install

[[email protected] redis-2.8.19]# cd utils/

[[email protected] utils]# ./install_server.sh

 

2.查看redis的监控端口

[[email protected] utils]# netstat -tlnp | grep redis

tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      30344/redis-server  

tcp        0      0 :::6379                     :::*                        LISTEN      30344/redis-server  

3.测试redis是否正常工作

[[email protected] utils]# cd ..

[[email protected] redis-2.8.19]# cd src/

[[email protected] src]# ./redis-cli -h 192.168.11.43 -p 6379

192.168.11.43:6379> ping

PONG

4.redis服务启动

[[email protected] ~]# cd /opt/software/

[[email protected] software]# ps -ef | grep redis

root      30344      1  0 10:43 ?        00:00:00 /usr/local/bin/redis-server *:6379              

root      30367   1628  0 10:51 pts/0    00:00:00 ./redis-cli -h 192.168.11.43 -p 6379

root      30370  28369  0 10:52 pts/2    00:00:00 grep redis

5.Redis的动态监控

[[email protected] software]# cd redis-2.8.19/src/

[[email protected] src]# ./redis-cli monitor

OK

6.Logstash结合redis工作

6.1确认redis服务是启动的

[[email protected] src]# ps -ef | grep redis

root      30344      1  0 10:43 ?        00:00:01 /usr/local/bin/redis-server *:6379              

root      30367   1628  0 10:51 pts/0    00:00:00 ./redis-cli -h 192.168.11.43 -p 6379

root      30379  28369  0 10:56 pts/2    00:00:00 grep redis

6.2启动redis动态监控

[[email protected] src]# ./redis-cli monitor

OK

6.3基于入口redis启动logstash

[[email protected] software]# vim logstash_to_redis.conf

[[email protected] software]# cat logstash_to_redis.conf

input { stdin { } }

output {

    stdout { codec => rubydebug }

    redis {

        host => '192.168.11.43'

        data_type => 'list'

        key => 'logstash:redis'

    }

}

[[email protected] software]# logstash agent -f logstash_to_redis.conf --verbose

 ELK日志分析

 

如果redis的监控上也有以上信息输出,表示Logstash和redis的结合是正常的。

ELK日志分析

三、Elasticsearch

1.安装Elasticsearch

[[email protected] software]# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz

[[email protected] software]# tar zxvf elasticsearch-1.7.2.tar.gz -C /usr/local

2.修改Elasticsearch配置文件elasticsearch.yml并且做以下修改:

[[email protected] software]# vim /usr/local/elasticsearch-1.7.2/config/elasticsearch.yml

discovery.zen.ping.multicast.enabled: false(关闭广播,如果局域网有机器开9300端口,服务会启动不了)

network.host: 192.168.11.43 (指定主机地址)

追加2行

http.cors.allow-origin: "/.*/"

http.cors.enabled: true

3.启动Elasticsearch服务

[[email protected] software]# /usr/local/elasticsearch-1.7.2/bin/elasticsearch  

注:日志会输出到stdout

[[email protected] software]# /usr/local/elasticsearch-1.7.2/bin/elasticsearch -d

注:表示以daemon方式启动

[[email protected] software]# nohup /usr/local/elasticsearch-1.7.2/bin/elasticsearch > /var/log/logstash.log 2>&1 &

4.查看Elasticsearch的监听端口

[[email protected] software]# netstat -tnlp| grep java

tcp        0      0 ::ffff:192.168.11.43:9200   :::*                        LISTEN      30514/java          

tcp        0      0 ::ffff:192.168.11.43:9201   :::*                        LISTEN      30588/java          

tcp        0      0 ::ffff:192.168.11.43:9202   :::*                        LISTEN      30645/java          

tcp        0      0 ::ffff:192.168.11.43:9300   :::*                        LISTEN      30514/java          

tcp        0      0 ::ffff:192.168.11.43:9301   :::*                        LISTEN      30588/java          

tcp        0      0 ::ffff:192.168.11.43:9302   :::*                        LISTEN      30645/java          

5.Elasticsearch和Logstash结合

Logstash的信息输出到Elasticsearch中

[[email protected] software]# vim logstash-elasticsearch.conf  

[[email protected] software]# cat logstash-elasticsearch.conf  

input { stdin {} }

output {

    elasticsearch { host => "192.168.11.43" }     

    stdout { codec=> rubydebug }

}

6.基于配置文件启动Logstash

[[email protected] software]# /usr/local/logstash-1.5.4/bin/logstash agent -f logstash-elasticsearch.conf

 ELK日志分析

 ELK日志分析

7.curl命令发送请求来查看Elasticsearch是否接收了数据

[[email protected] software]# curl http://192.168.11.43:9200/_search?pretty

{

  "took" : 161,

  "timed_out" : false,

  "_shards" : {

    "total" : 0,

    "successful" : 0,

    "failed" : 0

  },

  "hits" : {

    "total" : 0,

    "max_score" : 0.0,

    "hits" : [ ]

  }

}

8.安装Elasticsearch插件

Elasticsearch-kopf插件可以查询Elasticsearch中的数据,安装elasticsearch-kopf,只要在你安装Elasticsearch的目录中执行以下命令即可:

[[email protected] software]# cd /usr/local/elasticsearch-1.7.2/bin/

[[email protected] bin]# ./plugin install lmenezes/elasticsearch-kopf

 ELK日志分析

注:附手动下载

手动下载该软件,不通过插件安装命令...

cd /usr/local/elasticsearch-1.7.2/plugins

wget https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip

unzip master.zip

mv elasticsearch-kopf-master kopf

以上操作就完全等价于插件的安装命令

9.浏览器访问kopf页面访问elasticsearch保存的数据

[[email protected] bin]# netstat -tnlp | grep java

tcp        0      0 ::ffff:192.168.11.43:9200   :::*                        LISTEN      30514/java          

tcp        0      0 ::ffff:192.168.11.43:9201   :::*                        LISTEN      30588/java          

tcp        0      0 ::ffff:192.168.11.43:9202   :::*                        LISTEN      30645/java          

tcp        0      0 ::ffff:192.168.11.43:9300   :::*                        LISTEN      30514/java          

tcp        0      0 ::ffff:192.168.11.43:9301   :::*                        LISTEN      30588/java          

tcp        0      0 ::ffff:192.168.11.43:9302   :::*                        LISTEN      30645/java          

tcp        0      0 :::9303                     :::*                        LISTEN      30698/java          

 ELK日志分析

10.redis数据库中读取然后输出到elasticsearch中

[[email protected] software]# vim logstash-redis.conf

[[email protected] software]# cat logstash-redis.conf

input {

    redis {

        host => '192.168.11.43'

        data_type => 'list'

        port => "6379"

        key => 'logstash:redis'

        type => 'redis-input'

    }

}

output {

    elasticsearch {

        host => "192.168.11.43"

        codec => "json"

        protocol => "http"

    }

}

四、Kinaba

1.安装Kinaba

[[email protected] software]# wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz

[[email protected] software]# tar zxvf kibana-4.1.2-linux-x64.tar.gz -C /usr/local

2.修改kinaba配置文件kinaba.yml

[[email protected] software]# vim /usr/local/kibana-4.1.2-linux-x64/config/kibana.yml

elasticsearch_url: "http://192.168.11.43:9200"

3.启动kinaba

[[email protected] software]# /usr/local/kibana-4.1.2-linux-x64/bin/kibana

 

输出以上信息,表明kinaba成功。

kinaba默认监听在本地的5601端口上 。

4.浏览器访问kinaba

4.1 使用默认的logstash-*的索引名称,并且是基于时间的,点击“Create”即可

 ELK日志分析

 

看到如下界面说明索引创建完成。

 ELK日志分析

点击“Discover”,可以搜索和浏览Elasticsearch中的数据。

 ELK日志分析

分类:

技术点:

相关文章: