使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践--转载
原文地址：https://wsgzao.github.io/post/elk/
另外可以參考：https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04
前言
Elasticsearch + Logstash + Kibana（ELK）是一套開源的日志管理方案，分析網站的訪問情況時我們一般會借助Google/百度/CNZZ等方式嵌入JS做數據統計，但是當網站訪問異常或者被攻擊時我們需要在后臺分析如Nginx的具體日志，而Nginx日志分割/GoAccess/Awstats都是相對簡單的單節點解決方案，針對分布式集群或者數據量級較大時會顯得心有余而力不足，而ELK的出現可以使我們從容面對新的挑戰。
- Logstash：負責日志的收集，處理和儲存
- Elasticsearch：負責日志檢索和分析
- Kibana：負責日志的可視化
ELK(Elasticsearch + Logstash + Kibana)
更新記錄
2015年08月31日 - 初稿
閱讀原文 -?http://wsgzao.github.io/post/elk/
擴展閱讀
CentOS 7.x安裝ELK(Elasticsearch+Logstash+Kibana) -?http://www.chenshake.com/centos-install-7-x-elk-elasticsearchlogstashkibana/
Centos 6.5 安裝nginx日志分析系統 elasticsearch + logstash + redis + kibana -?http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=17291169&id=4898582
logstash-forwarder and grok examples -?https://www.ulyaoth.net/threads/logstash-forwarder-and-grok-examples.32413/
三斗室 -?http://chenlinux.com/
elastic -?https://www.elastic.co/guide
LTMP索引 -?http://wsgzao.github.io/index/#LTMP
組件預覽
JDK -?http://www.oracle.com/technetwork/java/javase/downloads/index.html
Elasticsearch -?https://www.elastic.co/downloads/elasticsearch
Logstash -?https://www.elastic.co/downloads/logstash
Kibana -?https://www.elastic.co/downloads/kibana
redis -?http://redis.io/download
設置FQDN
創建SSL證書的時候需要配置FQDN
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | #修改hostname cat /etc/hostname elk #修改hosts cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 10-10-87-19 10.10.87.19 elk.ooxx.com elk #刷新環境 hostname -F /etc/hostname #復查結果 hostname -f elk.ooxx.com hostname elk |
服務端
Java
| 1 2 3 4 5 6 7 8 9 | cat /etc/redhat-release CentOS release 6.5 (Final) yum install java-1.7.0-openjdk java -version java version "1.7.0_85" OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01) OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode) |
Elasticsearch
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | #下載安裝 wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm yum localinstall elasticsearch-1.7.1.noarch.rpm #啟動相關服務 service elasticsearch start service elasticsearch status #查看Elasticsearch的配置文件 rpm -qc elasticsearch /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/logging.yml /etc/init.d/elasticsearch /etc/sysconfig/elasticsearch /usr/lib/sysctl.d/elasticsearch.conf /usr/lib/systemd/system/elasticsearch.service /usr/lib/tmpfiles.d/elasticsearch.conf #查看端口使用情況 netstat -nltp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 1765/java tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 1765/java tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1509/sshd tcp 0 0 :::22 :::* LISTEN 1509/sshd #測試訪問 curl -X GET http://localhost:9200/ |
Kibana
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | #下載tar包 wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz #解壓 tar zxf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/ cd /usr/local/ mv kibana-4.1.1-linux-x64 kibana #創建kibana服務 vi /etc/rc.d/init.d/kibana #!/bin/bash ### BEGIN INIT INFO # Provides: kibana # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Runs kibana daemon # Description: Runs the kibana daemon as a non-root user ### END INIT INFO # Process name NAME=kibana DESC="Kibana4" PROG="/etc/init.d/kibana" # Configure location of Kibana bin KIBANA_BIN=/usr/local/kibana/bin # PID Info PID_FOLDER=/var/run/kibana/ PID_FILE=/var/run/kibana/$NAME.pid LOCK_FILE=/var/lock/subsys/$NAME PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN DAEMON=$KIBANA_BIN/$NAME # Configure User to run daemon process DAEMON_USER=root # Configure logging location KIBANA_LOG=/var/log/kibana.log # Begin Script RETVAL=0 if [ `id -u` -ne 0 ]; then echo "You need root privileges to run this script" exit 1 fi # Function library . /etc/init.d/functions start() { echo -n "Starting $DESC : " pid=`pidofproc -p $PID_FILE kibana` if [ -n "$pid" ] ; then echo "Already running." exit 0 else # Start Daemon if [ ! -d "$PID_FOLDER" ] ; then mkdir $PID_FOLDER fi daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 & sleep 2 pidofproc node > $PID_FILE RETVAL=$? [[ $? -eq 0 ]] && success || failure echo [ $RETVAL = 0 ] && touch $LOCK_FILE return $RETVAL fi } reload() { echo "Reload command is not implemented for this service." return $RETVAL } stop() { echo -n "Stopping $DESC : " killproc -p $PID_FILE $DAEMON RETVAL=$? echo [ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE } case "$1" in start) start ;; stop) stop ;; status) status -p $PID_FILE $DAEMON RETVAL=$? ;; restart) stop start ;; reload) reload ;; *) # Invalid Arguments, print the following message. echo "Usage: $0 {start|stop|status|restart}" >&2 exit 2 ;; esac #修改啟動權限 chmod +x /etc/rc.d/init.d/kibana #啟動kibana服務 service kibana start service kibana status |