Ubuntu 20.04 搜索引擎环境搭建 (PostgreSQL 12.3, Redis 6, ELK[Elasticsearch 7.8, Logstash 7.8, Kibana 7.8])
2021/02/04 更新redis安裝步驟
在軟件開發(fā)項(xiàng)目中,會(huì)遇到搜索引擎、日志收集等需求,本教程在阿里云選擇公共鏡像 Ubuntu 20.04 后,部署 apt-fast, java, axel, NodeJS, postgres, Redis, ELK(Elasticsearch, Logstash, Kibana, 均為開箱即用) 服務(wù)并設(shè)置開機(jī)自啟。阿里云 Ubuntu 20.04 系統(tǒng)鏡像下自帶Python3。
目錄
- 安裝
- 創(chuàng)建目錄
- 更新一波軟件
- 安裝 Apt-fast
- 安裝 Java
- 安裝 Axel
- 安裝 NodeJS
- 安裝 PostgreSQL
- 安裝 Redis
- 安裝 ELK
- 準(zhǔn)備安裝包(參考方式)
- 安裝 Elasticsearch
- 安裝 Logstash
- 安裝 Kibana
- 配置
- PostgreSQL
- Redis
- Elasticsearch
- Kibana
- 啟動(dòng)
- PostgreSQL
- Redis
- Elasticsearch
- Kibana
- 開機(jī)自啟
- Redis
- Elasticsearch
- Elasticsearch-head
- Kibana
- 其他
- Ubuntu如何刪除添加的倉庫
- PostgreSQL相關(guān)操作
- Linux相關(guān)操作
提醒:文件修改前請自己做好備份
軟件安裝在 /usr/local目錄下,日志文件放在**/var**目錄下
相關(guān)服務(wù)版本信息
| apt-fast | 2.0.3 (amd64) | apt-fast -v |
| Java | openjdk 14.0.1 | java --version |
| Axel | 2.17.5 (linux-gnu) | axel -V |
| Node.js | 14.7.0 | node -v |
| PostgreSQL | 12.3 | psql -V |
| Redis | 6.0.6 | redis-server -v |
| Elasticsearch | 7.8.0 | |
| Elasticsearch-head | 5.0.0 | |
| Elasticsearch-analysis-ik | 7.8.0 | |
| Logstash | 7.8.0 | |
| Kibana | 7.8.0 |
安裝
創(chuàng)建目錄
# 創(chuàng)建environment目錄 mkdir /usr/local/environment # 移動(dòng)到environment目錄 cd /usr/local/environment # 創(chuàng)建軟件目錄 mkdir redis elk elk/elasticsearch elk/logstash elk/kibana# 賦予讀寫權(quán)限 chmod 777 /usr/local/environment更新一波軟件
sudo apt-get upgrade安裝 Apt-fast
sudo apt install software-properties-common sudo add-apt-repository ppa:apt-fast/stable sudo apt-get -y install apt-fast # 安裝過程的配置,選擇apt-get, 最大線程數(shù)換成16,選擇no安裝 Java
sudo apt install openjdk-14-jdk-headless安裝 Axel
sudo apt-get install axel安裝 NodeJS
sudo apt install npm安裝 PostgreSQL
# Create the file repository configuration: sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'# Import the repository signing key: wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -# Update the package lists # Install the latest version of PostgreSQL. # If you want a specific version, use 'postgresql-12' or similar instead of 'postgresql': sudo apt-get update && sudo apt-get install postgresql #如果上面安裝速度太慢,使用apt-fast安裝 sudo apt-fast update sudo apt-fast install postgresql # 執(zhí)行npm install速度慢,更換淘寶鏡像 npm config set registry http://registry.npm.taobao.org安裝 Redis
# 進(jìn)入redis安裝目錄 cd /usr/local/environment/redis # 獲取redis安裝包 wget http://download.redis.io/releases/redis-6.0.6.tar.gz # 解壓(請使用對應(yīng)的文件名) tar -xvf redis-6.0.6.tar.gz # 創(chuàng)建軟鏈接(為了方便,非必須,如果沒有創(chuàng)建,接下來的涉及到redis路徑請用redis-6.0.5) # ln -s 原始目錄名 快速訪問目錄名 ln -s redis-6.0.6 redis # 進(jìn)入redis目錄,對解壓的Redis文件進(jìn)行編譯 cd redis make # 進(jìn)入src文件夾進(jìn)行Redis安裝 # ( 注意: 6.0.10 及以上的版本不需要運(yùn)行下面兩條命令) cd src make installredis安裝包下載鏈接從 redis 官網(wǎng)獲取。
安裝 ELK
準(zhǔn)備安裝包(參考方式)
# 進(jìn)入 ELK 目錄 cd /usr/local/environment/elk # 通過filezilla、FinalShell等工具將下載好的安裝包傳輸?shù)椒?wù)器官網(wǎng)/下載源:
Elasticsearch, elasticsearch-analysis-ik, elasticsearch-head, Logstash, Kibana
百度網(wǎng)盤:
鏈接: 百度網(wǎng)盤 提取碼: kdi9
PS: 7-zip可以(解壓ZIP文件然后)壓縮文件得到.tar.gz壓縮包
分別將[elasticsearch, elasticsearch-analysis, elasticsearch-head], [logstash], [kibana] 的壓縮包放在 elasticsearch, logstash, kibana 目錄下
安裝 Elasticsearch
# 進(jìn)入 elasticsearch 目錄 cd /usr/local/environment/elk/elasticsearch # 解壓安裝包 tar -zxvf elasticsearch-7.8.0-linux-x86_64.tar.gz tar -zxvf elasticsearch-head-5.0.0.tar.gz tar -zxvf elasticsearch-analysis-ik-7.8.0.tar.gz # 將 IK分詞器 文件移動(dòng)到elasticsearch的plugins目錄下 mv elasticsearch-analysis-ik-7.8.0 ./elasticsearch-7.8.0/plugins # 進(jìn)入 elasticsearch-head 目錄,安裝 elasticsearch-head 依賴包 cd elasticsearch-head-5.0.0 npm install安裝 Logstash
# 進(jìn)入 logstash 目錄 cd /usr/local/environment/elk/logstash # 解壓安裝包 tar -zxvf logstash-7.8.0.tar.gz安裝 Kibana
# 進(jìn)入 Kibana 目錄 cd /usr/local/environment/elk/kibana # 解壓安裝包 tar -zxvf kibana-7.8.0-linux-x86_64.tar.gz配置
PostgreSQL
創(chuàng)建用戶
# 修改數(shù)據(jù)庫用戶 sudo su - postgres # 連接數(shù)據(jù)庫(連接后命令輸入頭變成 postgres=# ) psql # 修改密碼 \password修改配置文件 postgresql.conf 允許所有ip訪問
listen_addresses = '*'修改認(rèn)證權(quán)限配置文件 pg_hba.conf ,添加一下配置,并將所有的"peer"改為"md5"(密碼認(rèn)證)
# IPv4 local connections: host all all 0.0.0.0/0 md5(我的配置文件在/etc/postgresql/12/main目錄下)
Redis
# 在/var/log目錄下創(chuàng)建redis文件夾,存放redis日志文件 mkdir /var/log/redis # 進(jìn)入redis安裝目錄,創(chuàng)建conf、data文件夾 cd /usr/local/environment/redis/redis mkdir data conf # 進(jìn)入conf目錄,編寫配置文件 redis-端口號(hào).conf # 例:redis-6379.conf cd conf touch redis-6379.conf(除了帶注釋的行,其他均為默認(rèn)設(shè)置,見redis目錄下的redis.conf文件)
# 綁定的ip,請根據(jù)需求自行更換 # bind 127.0.0.1 # 遠(yuǎn)程訪問 protected-mode no protected-mode no # 端口設(shè)置 port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 # 以守護(hù)進(jìn)程的方式啟動(dòng),redis以服務(wù)的形式存在,日志不再打印到命令窗口中 daemonize yes supervised no pidfile /var/run/redis-6379.pid loglevel notice # 日志文件,使用絕對路徑,防止出問題 logfile "/var/log/redis/6379.log" databases 16 always-show-logo yes save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb rdb-del-sync-files no # 當(dāng)前服務(wù)文件保存位置,包含日志文件、持久化文件等,使用絕對路徑,防止出問題 dir /var/log/redis replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-diskless-load disabled repl-disable-tcp-nodelay no replica-priority 100 acllog-max-len 128 lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no lazyfree-lazy-user-del no appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes jemalloc-bg-thread yes # 內(nèi)存到達(dá)上限之后的處理策略 maxmemory-policy noeviction # 1. volatile-lru:對設(shè)置了過期時(shí)間的key執(zhí)行LRU逐出策略(默認(rèn)值) # 2. allkeys-lru : 對所有key使用LRU逐出策略 # 3. volatile-random:隨機(jī)刪除即將過期key(LFU) # 4. allkeys-random:隨機(jī)刪除 # 5. volatile-ttl : 刪除即將過期的 # 6. noeviction : 永不過期,返回錯(cuò)誤Elasticsearch
# 進(jìn)入Elasticsearch安裝目錄 cd /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0 # 創(chuàng)建文件夾data、logs,存放elasticsearch數(shù)據(jù),日志 mkdir data logs # 進(jìn)入Elasticsearch配置文件目錄 cd /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0/config # 編輯該目錄下的elasticsearch.yml文件,在文件末尾加上下面配置(或者取消注釋后修改) #集群名稱 cluster.name: SUSTech-Town-ES#節(jié)點(diǎn)名稱 node.name: node-st#數(shù)據(jù)和日志的存儲(chǔ)目錄(若安裝路徑不同, 請注意更改) path.data: /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0/data path.logs: /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0/logs#設(shè)置綁定的ip,0.0.0.0代表任意ip訪問 network.host: 0.0.0.0#端口 http.port: 9200#設(shè)置在集群中的所有節(jié)點(diǎn)名稱 cluster.initial_master_nodes: ["node-st"]# 配置elasticsearch允許跨域訪問 http.cors.enabled: true http.cors.allow-origin: "*"修改系統(tǒng)的進(jìn)程內(nèi)存限制,否則啟動(dòng)es會(huì)報(bào)錯(cuò)
/etc/security目錄下的***limits.conf***文件,更新/添加以下數(shù)據(jù)
* soft nofile 65536 * hard nofile 131072 * soft nproc 4096 * hard nproc 4096/etc目錄下的***sysctl.conf***文件,添加以下配置
# elasticsearch vm.max_map_count=262145添加完成并保存后執(zhí)行
sysctl -pelasticsearch不允許以root用戶啟動(dòng),故新建其他用戶來啟動(dòng)
# 創(chuàng)建用戶組es groupadd elk # 創(chuàng)建新用戶elk, -g elk 設(shè)置其用戶組為elk, -p 123456 設(shè)置其密碼123456 useradd elk -g elk -p 123456 # 更改 elk 文件夾及內(nèi)部文件的所屬用戶及組為 elk:elk chown -R elk:elk /usr/local/environment/elkKibana
修改/usr/local/environment/elk/kibana/kibana-7.8.0-linux-x86_64/config目錄下的***kibana.yml***文件
server.port: 5601 # 允許所有ip訪問 server.host: "0.0.0.0" server.name: "SUSTechTown" elasticsearch.hosts: ["http://localhost:9200"]# 按需開啟 #elasticsearch.username: "kibana_system" #elasticsearch.password: "pass"pid.file: /var/run/kibana.pid# 語言中文 i18n.locale: "zh-CN"# 加密,注意:參數(shù)值至少32位,否則啟動(dòng)會(huì)報(bào)錯(cuò)提示 xpack.encryptedSavedObjects.encryptionKey: SUSTechTownencryptedSavedObjects0123456789 xpack.security.encryptionKey: SUSTechTownencryptionKeysecurity01234567890 xpack.reporting.encryptionKey: SUSTechTownencryptionKeyreporting0123456789 # 給elasticsearch及其以下子目錄賦予讀寫權(quán)限 chmod -R 777 /usr/local/environment啟動(dòng)
PostgreSQL
命令行的各個(gè)參數(shù)解釋說明:
- -U username 用戶名,默認(rèn)值postgres
- -d dbname 要連接的數(shù)據(jù)庫名,默認(rèn)值postgres。如果單指定-U,沒指定-d參數(shù),則默認(rèn)訪問與用戶名名稱相同的數(shù)據(jù)庫。
- -h hostname 主機(jī)名,默認(rèn)值localhost
- -p port 端口號(hào),默認(rèn)值5432
Redis
# 進(jìn)入redis配置文件目錄 cd /usr/local/environment/redis/redis/conf # 指定配置文件開啟redis服務(wù) redis-server redis-6379.confPS: /usr/local/environment/redis/redis/data 目錄下的日志 6379.log 如有 WARING 則按照其提示操作
(最初安裝沒有配置是有WARING的,操作提示很明了)
Elasticsearch
# 進(jìn)入elasticsearch安裝目錄 cd /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0 # 切換用戶 su elk # 啟動(dòng)elasticsearch ./bin/elasticsearch # 或者 后臺(tái)啟動(dòng) ./bin/elasticsearch -d # 啟動(dòng)elasticsearch-head(必須先啟動(dòng)elasticsearch) cd /usr/local/environment/elk/elasticsearch/elasticsearch-head-5.0.0 npm run start # 或者 后臺(tái)啟動(dòng) nohup npm run start &如果出現(xiàn)一下報(bào)錯(cuò),說明IK分詞器插件安裝不正確
java.lang.IllegalStateException: Could not load plugin descriptor for plugin directory [elasticsearch-analysis-ik-7.8.0] Likely root cause: java.nio.file.NoSuchFileException: /usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0/plugins/elasticsearch-analysis-ik-7.8.0/plugin-descriptor.properties檢查elasticsearch-analysis-ik-7.8.0文件夾(或者ik分詞器壓縮包)下是否存在plugin-descriptor.properties文件
Kibana
# 進(jìn)入kibana安裝目錄 cd /usr/local/environment/elk/kibana/kibana-7.8.0-linux-x86_64 # 切換用戶 su elk # 啟動(dòng)kibana ./bin/kibana # 或者 后臺(tái)啟動(dòng) nohup ./bin/kibana &若出現(xiàn)一下報(bào)錯(cuò)無法啟動(dòng)
FATAL Error: EACCES: permission denied, open '/var/run/kibana.pid'則到/var/run目錄下新建文件kibana.pid或執(zhí)行一下命令 touch /var/run/kibana.pid
開機(jī)自啟
配置使用 systemctl 管理 redis, elasticsearch, elasticsearch-head, kibana
# 進(jìn)入腳本文件目錄 cd /etc/init.d # 創(chuàng)建腳本文件 touch redis elasticsearch elasticsearch-head kibanaRedis
#!/bin/bash #chkconfig: 22345 10 90 #description: Start and Stop redis# redis安裝目錄 REDIS_PATH=/usr/local/environment/redis/redis-6.0.6 # 啟動(dòng)端口號(hào) REDIS_PORT=6379 EXEC=$REDIS_PATH/src/redis-server CLIEXEC=$REDIS_PATH/src/redis-cliPIDFILE=/var/run/redis-6379.pid # redis配置文件路徑 CONF="$REDIS_PATH/conf/redis-6379.conf"case "$1" instart)if [ -f $PIDFILE ];thenecho "$PIDFILE exists,process is already running or crashed"elseecho "Starting Redis server..."$EXEC $CONFfi;;stop)if [ ! -f $PIDFILE ];thenecho "$PIDFILE does not exist,process is not running"elsePID=$(cat $PIDFILE)echo "Stopping..."$CLIEXEC -p $REDIS_PORT shutdownwhile [ -x /proc/${PID} ]doecho "Waiting for Redis to shutdown..."sleep 1doneecho "Redis stopped"fi;;restart)"$0" stopsleep 3"$0" start;;*)echo "Please use start or stop or restart as first argument";; esac # 修改文件權(quán)限 chmod +x /etc/init.d/redisElasticsearch
#!/bin/sh #chkconfig: 2345 80 05#description: elasticsearch# 用于啟動(dòng)elasticsearch的用戶 ES_USER=elk # elasticsearch安裝目錄 ES_PATH=/usr/local/environment/elk/elasticsearch/elasticsearch-7.8.0case "$1" instart)su $ES_USER << !$ES_PATH/bin/elasticsearch -d; !echo "elasticsearch startup";;stop)es_pid=`ps aux|grep elasticsearch | grep -v 'grep elasticsearch' | awk '{print $2}'`kill -9 $es_pidecho "elasticsearch stopped";;restart)es_pid=`ps aux|grep elasticsearch | grep -v 'grep elasticsearch' | awk '{print $2}'`kill -9 $es_pidecho "elasticsearch stopped"sleep 3su $ES_USER << !$ES_PATH/bin/elasticsearch -d; !echo "elasticsearch restarted";;*)echo "Usage: $0 {start|stop|restart|force-reload|status}";; esac # 修改文件權(quán)限 chmod +x /etc/init.d/elasticsearchElasticsearch-head
#!/bin/sh#chkconfig: 2345 80 05 #description elasticsearch-head# elasticsearch-head 啟動(dòng)端口 ES_HEAD_HOST=9100 # elasticsearch-head 安裝路徑 ES_HEAD_PATH=/usr/local/environment/elk/elasticsearch/elasticsearch-head-5.0.0 # 用于啟動(dòng)elasticsearch-head的用戶 ES_HEAD_USER=elk # nodejs 安裝路徑(可以通過此命令獲得: which node) export NODE_PATH=/usr/bin/node export PATH=$PATH:$NODE_PATH/bincase "$1" instart)su $ES_HEAD_USER<<!cd $ES_HEAD_PATH;nohup npm run start >$ES_HEAD_PATH/nohup.out 2>&1 & !echo "elasticsearch-head startup";;stop)es_head_pid=`lsof -i :9100 | awk '{print $2}' | sed -n '2p'`kill -9 $es_head_pidecho "elasticsearch-head stopped";;restart)es_head_pid=`lsof -i :9100 | awk '{print $2}' | sed -n '2p'`kill -9 $es_head_pidecho "elasticsearch-head stopped"sleep 3su $ES_HEAD_USER<<!cd $ES_HEAD_PATH;nohup npm run start >$ES_HEAD_PATH/nohup.out 2>&1 & !echo "elasticsearch-head restarted";;*)echo "Usage: $0 {start|stop|restart|force-reload|status}"esac # 修改文件權(quán)限 chmod +x /etc/init.d/elasticsearch-headKibana
#!/bin/sh # # /etc/init.d/kibana4_init -- startup script for kibana4 ### BEGIN INIT INFO # Provides: kibana4_init # Required-Start: $network $remote_fs $named # Required-Stop: $network $remote_fs $named # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Starts kibana4_init # Description: Starts kibana4_init using start-stop-daemon ### END INIT INFO#!/bin/sh #chkconfig: 2345 80 05#description: kibana# 用于啟動(dòng) kibana 的用戶 KIBANA_USER=elk # kibana安裝目錄 KIBANA_PATH=/usr/local/environment/elk/kibana/kibana-7.8.0-linux-x86_64case "$1" instart)su $KIBANA_USER << !nohup $KIBANA_PATH/bin/kibana & !echo "kibana startup";;stop)kibana_pid=`ps aux|grep kibana | grep -v 'grep kibana' | awk '{print $2}'`kill -9 $kibana_pidecho "kibana stopped";;restart)kibana_pid=`ps aux|grep kibana | grep -v 'grep kibana' | awk '{print $2}'`kill -9 $kibana_pidecho "kibana stopped"su $KIBANA_USER << !nohup $KIBANA_PATH/bin/kibana & !echo "kibana restarted";;*)echo "Usage: $0 {start|stop|restart|force-reload|status}";; esac # 修改文件權(quán)限 chmod +x /etc/init.d/kibana此時(shí)可以通過以下命令啟動(dòng)、停止redis服務(wù),查看服務(wù)狀態(tài)
systemctl start redis systemctl stop redis systemctl status redis設(shè)置開機(jī)啟動(dòng)
打開 /lib/systemd/system 目錄下的 rc.local.service 文件,末尾加入以下內(nèi)容
[Install] WantedBy=multi-user.target Alias=rc-local.service在 /etc 目錄下新建 rc.loacl 文件并加上權(quán)限
sudo touch /etc/rc.local sudo chmod +x /etc/rc.local編寫 rc.loacl 文件內(nèi)容
#!/bin/sh echo never > /sys/kernel/mm/transparent_hugepage/enabled# 服務(wù)自啟 systemctl start redis systemctl start elasticsearch systemctl start elasticsearch-head systemctl start kibana exit 0重啟命令測試
reboot其他
Ubuntu如何刪除添加的倉庫
我們在安裝某些軟件的時(shí)候需要提前設(shè)置他的倉庫,但是裝好后每次運(yùn)行apt-get update 都會(huì)從重新去掃描這個(gè)倉庫,沒報(bào)錯(cuò)還好,但有些倉庫會(huì)報(bào)錯(cuò)。因此我們需要?jiǎng)h除沒用的倉庫。
方法:
1.用代碼,這里不寫了。
2.不用代碼,直接在/etc/apt/sources.list.d/文件夾里面找到需要?jiǎng)h除的倉庫
PostgreSQL相關(guān)操作
# 查看PostgreSQL狀態(tài) service postgresql status # 重啟PostgreSQL service postgresql restart# client命令行登錄(下列參數(shù)自己替換) psql -U {USERNAME} -d {DATABASE} -h {HOST} -p {PORT}# 登錄數(shù)據(jù)庫之后的操作 # 建立新的數(shù)據(jù)庫用戶 create user zhangsan with password '123456'; # 為新用戶建立數(shù)據(jù)庫 create database testdb owner zhangsan; # 把新建的數(shù)據(jù)庫權(quán)限賦予新用戶 grant all privileges on database testdb to zhangsan;#創(chuàng)建超級(jí)用戶 create user checker with superuser; \password checker;Linux相關(guān)操作
# 查看后臺(tái) redis 進(jìn)程 ps -ef | grep redis # 查看后臺(tái) elasticsearch 進(jìn)程 ps -ef | grep elasticsearch # 查看 nginx 安裝目錄 nginx -t我的小站、Github
總結(jié)
以上是生活随笔為你收集整理的Ubuntu 20.04 搜索引擎环境搭建 (PostgreSQL 12.3, Redis 6, ELK[Elasticsearch 7.8, Logstash 7.8, Kibana 7.8])的全部內(nèi)容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: Windows 10 下 VS2017(
- 下一篇: CSDN在文章头部添加目录