centos6.5下安装配置ELK及收集nginx日志
生活随笔
收集整理的這篇文章主要介紹了
centos6.5下安装配置ELK及收集nginx日志
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
Elasticsearch 是個開源分布式搜索引擎，它的特點有：分布式，零配置，自動發現，索引自動分片，索引副本機制，restful風格接口，多數據源，自動搜索負載等。
Logstash 是一個完全開源的工具，他可以對你的日志進行收集、分析，并將其存儲供以后使用（如，搜索）
kibana 也是一個開源和免費的工具，他Kibana可以為 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以幫助您匯總、分析和搜索重要數據日志。
環境：192.168.50.119：ELK+Nginx
? ? 192.168.50.120：Redis+Logstash
架構圖
部署流程：
192.168.50.119 ELK服務器
1.安裝JDK
Logstash的運行依賴于Java運行環境， Logstash 1.5以上版本不低于java 7推薦使用最新版本的Java，我這里使用了1.8版本
tar?-zxf?jdk-8u45-linux-x64.tar.gz?-C?/usr/local/ vim?/etc/profile??#設置環境變量 export?JAVA_HOME=/usr/local/jdk1.8.0_45 export?PATH=$PATH:$JAVA_HOME/bin export?CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH source?/etc/profile??#使環境變量生效驗證是否安裝成功
2.安裝Logstash（日志收集、分析，并將其存儲供以后使用）
wget?https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz tar?–zxf?logstash-2.4.0.tar.gz?-C?/usr/local/驗證logstash是否安裝成功
[root@localhost?~]#?/usr/local/logstash-2.4.0/bin/logstash?-e?'input?{?stdin?{?}?}?output?{?stdout?{}?}' Settings:?Default?pipeline?workers:?1 Logstash?startup?completed 等待輸入:hello?world 2016-11-28T20:32:07.853Z?localhost.localdomain?hello?world我們可以看到，我們輸入什么內容logstash按照某種格式輸出，其中-e參數參數允許Logstash直接通過命令行接受設置。
這點尤其快速的幫助我們反復的測試配置是否正確而不用寫配置文件。使用CTRL-C命令可以退出之前運行的Logstash。
3.部署nginx并收集日志
yum?-y?install?nginx 設置nginx的log?格式 vim?/etc/nginx/nginx.conflog_format?main?'$remote_addr?-?$remote_user?[$time_local]?"$request"?''$status?$body_bytes_sent?"$http_referer"?''"$http_user_agent"?$http_x_forwarded_for?$request_length?$msec?$connection_requests?$request_time';啟動nginx
service?nginx?startmkdir?/usr/local/logstash-2.4.0/conf/???#創建logstash配置目錄 定義logstash配置文件，用來收集nginx日志 [root@localhost?conf]#?cat?logstash_nginx.conf? input?{file?{path?=>?["/var/log/nginx/access.log"]type?=>?"nginx_log"} } output?{redis{host?=>?"192.168.50.120"key?=>?'logstash-redis'data_type?=>?'list'}stdout?{ codec?=>?rubydebug} }4.安裝部署redis ? ?
192.168.50.120 服務器
yum?-y?install?redis vim?/etc/redis.conf bind?192.168.50.120啟動
service?redis?start5.啟動Logstash
[root@localhost?conf]#?/usr/local/logstash-2.4.0/bin/logstash?-f?./logstash_nginx.conf??--configtest???#檢查配置文件 Configuration?OK[root@localhost?conf]#?/usr/local/logstash-2.4.0/bin/logstash?agent??-f?./logstash_nginx.conf??????????#將日志信息輸出到redis服務器 Settings:?Default?pipeline?workers:?1 Logstash?startup?completed {"message"?=>?"192.168.50.114?-?-?[29/Nov/2016:00:58:43?+0800]?\"GET?/?HTTP/1.1\"?304?0?\"-\"?\"Mozilla/5.0?(Windows?NT?6.1;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/54.0.2840.99?Safari/537.36\"?\"-\"","@version"?=>?"1","@timestamp"?=>?"2016-11-28T18:55:49.587Z","path"?=>?"/var/log/nginx/access.log","host"?=>?"localhost.localdomain","type"?=>?"nginx_log" } {"message"?=>?"192.168.50.114?-?-?[29/Nov/2016:00:58:43?+0800]?\"GET?/nginx-logo.png?HTTP/1.1\"?304?0?\"http://192.168.50.119/\"?\"Mozilla/5.0?(Windows?NT?6.1;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/54.0.2840.99?Safari/537.36\"?\"-\"","@version"?=>?"1","@timestamp"?=>?"2016-11-28T18:55:49.590Z","path"?=>?"/var/log/nginx/access.log","host"?=>?"localhost.localdomain","type"?=>?"nginx_log" } {"message"?=>?"192.168.50.114?-?-?[29/Nov/2016:00:58:43?+0800]?\"GET?/poweredby.png?HTTP/1.1\"?304?0?\"http://192.168.50.119/\"?\"Mozilla/5.0?(Windows?NT?6.1;?Win64;?x64)?AppleWebKit/537.36?(KHTML,?like?Gecko)?Chrome/54.0.2840.99?Safari/537.36\"?\"-\"","@version"?=>?"1","@timestamp"?=>?"2016-11-28T18:55:49.590Z","path"?=>?"/var/log/nginx/access.log","host"?=>?"localhost.localdomain","type"?=>?"nginx_log" }6.安裝部署Elasticsearch
192.168.50.119 ELK服務器
創建安裝用戶
groupadd?elk useradd?es?-g?elktar?-xf?elasticsearch-2.2.0.tar.gz?-C?/usr/local/ vim?/usr/local/elasticsearch-2.2.0/config/elasticsearch.ymlnetwork.host:?192.168.50.119???#?端口綁定ip地址http.port:?9200啟動
這里遇到一個坑：es用戶默認是不能用root用戶啟動的。所以要切到普通用戶啟動
chown?-R?es.elk?/usr/local/elasticsearch-2.2.0 su?-?es nohup??/usr/local/elasticsearch-2.2.0/bin/elasticsearch?>/usr/local/elasticsearch-2.2.0/nohub?&[root@localhost?ELK]#?netstat?-tunpl?|?grep?9200 tcp????????0??????0?::ffff:192.168.50.119:9200??:::*????????????????????????LISTEN??????2183/java[root@localhost?ELK]#?curl?http://192.168.50.119:9200???#查看狀態 {"name"?:?"Blood?Brothers","cluster_name"?:?"elasticsearch","version"?:?{"number"?:?"2.2.0","build_hash"?:?"8ff36d139e16f8720f2947ef62c8167a888992fe","build_timestamp"?:?"2016-01-27T13:32:39Z","build_snapshot"?:?false,"lucene_version"?:?"5.4.1"},"tagline"?:?"You?Know,?for?Search" }安裝kopf和head插件
[root@localhost?conf]#?cd?/usr/local/elasticsearch-2.2.0/bin/ [root@localhost?bin]#?./plugin??install?lmenezes/elasticsearch-kopf ->?Installing?lmenezes/elasticsearch-kopf... Trying?https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip?... Downloading?............................................................?DONE Verifying?https://github.com/lmenezes/elasticsearch-kopf/archive/master.zip?checksums?if?available?... NOTE:?Unable?to?verify?checksum?for?downloaded?plugin?(unable?to?find?.sha1?or?.md5?file?to?verify) Installed?kopf?into?/usr/local/elasticsearch-2.2.0/plugins/kopf[root@localhost?bin]#?./plugin?install?mobz/elasticsearch-head ->?Installing?mobz/elasticsearch-head... Trying?https://github.com/mobz/elasticsearch-head/archive/master.zip?... Downloading?.........................................................DONE NOTE:?Unable?to?verify?checksum?for?downloaded?plugin?(unable?to?find?.sha1?or?.md5?file?to?verify) Installed?head?into?/usr/local/elasticsearch-2.2.0/plugins/head7.安裝kibana?
192.168.50.119 ELK服務器
安裝
[root@localhost?ELK]#?tar?-xf?kibana-4.4.0-linux-x64.tar.gz??-C?/usr/local/ [root@localhost?ELK]#?cd?/usr/local/kibana-4.4.0-linux-x64/配置
[root@localhost?kibana-4.4.0-linux-x64]#?vim?config/kibana.yml elasticsearch.url:?"http://192.168.50.119:9200" server.port:?5601 server.host:?"0.0.0.0"啟動
[root@localhost?kibana-4.4.0-linux-x64]#?nohup??/usr/local/kibana-4.4.0-linux-x64/bin/kibana?>?/usr/local/kibana-4.4.0-linux-x64/nohub.out?&[root@localhost?ELK]#?netstat?-tunpl?|?grep?5601 tcp????????0??????0?0.0.0.0:5601????????????????0.0.0.0:*瀏覽器訪問http://192.168.50.119:5601/
8.安裝logstash-server服務器?
192.168.50.120 ?服務器
安裝jdk和logstash
將redis 中的數據發送到elasticsearch中
[root@localhost?conf]#?cat?logstash_server.conf? input?{redis?{port?=>?"6379"host?=>?"192.168.50.120"data_type?=>?"list"key?=>?"logstash-redis"type?=>?"redis-input"} } output?{elasticsearch?{hosts?=>?"192.168.50.119"index?=>?"logstash-%{+YYYY.MM.dd}"} }9.在Kibanda上創建nginx日志監控視圖
es常規操作
es?健康狀態 [root@localhost?~]#?curl??http://192.168.50.119:9200/_cat/health?v epoch??????timestamp?cluster???????status?node.total?node.data?shards?pri?relo?init?unassign?pending_tasks?max_task_wait_time?active_shards_percent? 1480345315?23:01:55??elasticsearch?yellow??????????1?????????1??????6???6????0????0????????6?????????????0??????????????????-?????????????????50.0%? health?的狀態包括：green,?yellow,?red.? 列出節點 [root@localhost?~]#?curl??http://192.168.50.119:9200/_cat/nodes?v host???????????ip?????????????heap.percent?ram.percent?load?node.role?master?name??????????? 192.168.50.119?192.168.50.119????????????8??????????99?0.00?d?????????*??????Blood?Brothers? 列出索引 [root@localhost?~]#?curl??http://192.168.50.119:9200/_cat/indices?v health?status?index???????????????pri?rep?docs.count?docs.deleted?store.size?pri.store.size? yellow?open???.kibana???????????????1???1??????????2????????????0??????5.6kb??????????5.6kb? yellow?open???logstash-2016.11.28???5???1??????????1????????????0??????4.9kb??????????4.9kb參考地址：
http://liumissyou.blog.51cto.com/4828343/1850973? ?? ? ? ? ? ? ? ? http://ckl893.blog.51cto.com/8827818/1772287? ?
轉載于:https://blog.51cto.com/thedream/1878971
總結
以上是生活随笔為你收集整理的centos6.5下安装配置ELK及收集nginx日志的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： WEB项目 后台接收前端数组
- 下一篇： DB1：数据库的创建和文件的修改