linux 内存耗尽重启,【server故障】linux下JVM内存耗尽故障
– 環境:linux RHL 7.5
– Tableau Server:2019.1
今天客戶咨詢一個問題,在訪問8850后臺時,提示出現以下錯誤:
Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.There was an unexpected error (type=Internal Server Error, status=500).
com.tableausoftware.tabadmin.webapp.exceptions.RestException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
1、故障排查和描述
第一時間,我讓客戶使用tsm命令查看了server的狀態,顯示異常如下:
我們看到上面的關鍵提示是Native memory allocation (mmap) failed to map,通過這里我們可以看到有用的線索,同時根據log日志進一步查找原因。
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 702021632 bytes for committing reserved memory.
# Possible reasons:
# ? The system is out of physical RAM or swap space
# ? In 32 bit mode, the process size limit was hit
# Possible solutions:
# ? Reduce memory load on the system
# ? Increase physical memory or swap space
# ? Check if swap backing store is full
# ? Use 64 bit Java on a 64 bit OS
# ? Decrease Java heap size (-Xmx/-Xms)
# ? Decrease number of Java threads
# ? Decrease Java thread stack sizes (-Xss)
# ? Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# ?Out of Memory Error (os_linux.cpp:2640), pid=6197, tid=0x00007fca569c1700
#
# JRE version: ?(8.0_181-b13) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
#
2、故障解決
此類問題的首選推薦是重啟server,方法是使用tsm restart命令。不過重啟后,所有進程依然無法使用,tsm狀態顯示error,后臺服務均未啟動。
此時需要進一步查看日志文件,Tableau server的日志保存在以下路徑中:
/var/opt/tableau/tableau_server/data/tabsvc/logs/
首先查看負責tsm命令的tabadmincontroller_node1-0日志文件,發現一切正常,沒有error類的提示;由于上面提到了zookeeper,之后查看負責協調服務的日志,appzookeeper_node1-0.log 日志,反復記錄的error問題為:
2019-07-14 21:01:37.130 +0800 29300 main : ERROR org.apache.zookeeper.server.quorum.QuorumPeerConfig – Invalid configuration, only one server specified (ignoring)
既然是協調服務的故障,可能與系統的底層故障有關,因此建議顧客重啟操作系統服務器,
重啟了操作系統之后,tsm方才正常。
正在通過工程師排查進一步的原因,工程師反饋后會補充。
Jul 14, 2019
吳玉朋
Tableau partner
18615511300
在 2019年7月14日,下午11:16,Tableau Technical Support Case Replies 寫道:
員
總結
以上是生活随笔為你收集整理的linux 内存耗尽重启,【server故障】linux下JVM内存耗尽故障的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: 微软2016校园招聘4月在线笔试 hih
- 下一篇: 现代版的———项链 :( 转