當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Elasticsearch聚合查询案例分享

發布時間：2025/3/18 编程问答 20 豆豆

生活随笔收集整理的這篇文章主要介紹了 Elasticsearch聚合查询案例分享小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

為什么80%的碼農都做不了架構師？>>> ??

Elasticsearch聚合查詢案例分享?

?

1.案例介紹?

本文包含三個案例:

案例1：統計特定時間范圍內每個應用的總訪問量、訪問成功數、訪問失敗數,每個應用請求響應時間分段統計（1秒內，1-3秒，3-5秒，5秒以上）

案例2：簡單的term統計

案例3：簡單的cardinality統計

2.準備工作?

參考文檔《高性能elasticsearch ORM開發庫使用介紹》中的第1章節和第2章節，在自己的工程中導入bboss es依賴包和配置es參數?

3.案例

3.1 案例1?

3.1.1?定義統計dsl?

在源碼目錄下新建文件esmapper/estrace/ESTracesMapper.xml，內容如下?

<properties><property name="applicationSumStatic"><![CDATA[{"query": {"bool": {"filter": [#if($channelApplications && $channelApplications.size() > 0){"terms": {"applicationName.keyword": [ ##指定并統計多個應用的數據#foreach($application in $channelApplications)#if($velocityCount > 0),#end $application.applicationName#end]}},#end{"range": {"startTime": {"gte": #[startTime],##統計開始時間"lt": #[endTime] ##統計截止時間}}}]}},"size":0,"aggs": {"applicationsums": {"terms": {"field": "applicationName.keyword",##按應用名稱進行統計計數"size":10000},"aggs":{"successsums" : {"terms" : {"field" : "err" ##按err標識統計每個應用的成功數和失敗數，0標識成功，1標識失敗}},"elapsed_ranges" : {"range" : {"field" : "elapsed", ##按響應時間分段統計"keyed" : true,"ranges" : [{ "key" : "1秒", "to" : 1000 },{ "key" : "3秒", "from" : 1000, "to" : 3000 },{ "key" : "5秒", "from" : 3000, "to" : 5000 },{ "key" : "5秒以上", "from" : 5000 }]}}}}}}]]></property> </properties>

3.1.2 編寫統計dao及統計方法?

Java代碼?

public class TraceESDao { public List<ApplicationStatic> getApplicationSumStatic(TraceExtraCriteria traceExtraCriteria){ClientInterface clientUtil= ElasticSearchHelper.getConfigRestClientUtil("esmapper/estrace/ESTracesMapper.xml");//返回json統計報文，調試用，一遍根據json報文組裝統計結果列表 // String response = clientUtil.executeRequest("trace-*/_search", // "applicationSumStatic",traceExtraCriteria);//根據條件進行統計，在對象traceExtraCriteria中指定開始時間和結束時間MapRestResponse restResponse = clientUtil.search("trace-*/_search","applicationSumStatic",traceExtraCriteria);//組裝統計結果//獲取應用統計列表，包含每個應用的名稱、總訪問量以及成功數和失敗數List<Map<String,Object>> appstatics = (List<Map<String,Object>>)restResponse.getAggBuckets("applicationsums");if(appstatics != null && appstatics.size() > 0) {List<ApplicationStatic> applicationStatics = new ArrayList<ApplicationStatic>(appstatics.size());ApplicationStatic applicationStatic = null;for (int i = 0; i < appstatics.size(); i++) {applicationStatic = new ApplicationStatic();Map<String, Object> map = appstatics.get(i);//應用名稱String appName = (String) map.get("key");applicationStatic.setApplicationName(appName);//應用總訪問量Long totalsize = ResultUtil.longValue( map.get("doc_count"),0l);applicationStatic.setTotalSize(totalsize);//獲取成功數和失敗數List<Map<String, Object>> appstatic = (List<Map<String, Object>>)ResultUtil.getAggBuckets(map, "successsums");/**"buckets": [{"key": 0,"doc_count": 30}]*///key 0Long success = 0l;//成功數Long failed = 0l;//失敗數for (int j = 0; j < appstatic.size(); j++) {Map<String, Object> stats = appstatic.get(j);Integer key = (Integer) stats.get("key");//成功和錯誤標識if (key == 0)//成功success = ResultUtil.longValue( stats.get("doc_count"),0l);else if (key == 1)//失敗failed = ResultUtil.longValue( stats.get("doc_count"),0l);}applicationStatic.setSuccessCount(success);applicationStatic.setFailCount(failed);List<ApplicationPeriodStatic> applicationPeriodStatics = new ArrayList<ApplicationPeriodStatic>(4);ApplicationPeriodStatic applicationPeriodStatic = null;//獲取響應時間分段統計信息Map<String, Map<String, Object>> appPeriodstatic = (Map<String, Map<String, Object>>)ResultUtil.getAggBuckets(map, "elapsed_ranges");//1秒Map<String, Object> period = appPeriodstatic.get("1秒");applicationPeriodStatic = new ApplicationPeriodStatic();applicationPeriodStatic.setPeriod("1秒");applicationPeriodStatic.setDocCount(ResultUtil.longValue(period.get("doc_count"),0l));applicationPeriodStatic.setTo(ResultUtil.intValue(period.get("to"),1000));applicationPeriodStatics.add(applicationPeriodStatic);//3秒period = appPeriodstatic.get("3秒");applicationPeriodStatic = new ApplicationPeriodStatic();applicationPeriodStatic.setPeriod("3秒");applicationPeriodStatic.setDocCount(ResultUtil.longValue(period.get("doc_count"),0l));applicationPeriodStatic.setFrom(ResultUtil.intValue(period.get("from"),1000));applicationPeriodStatic.setTo(ResultUtil.intValue(period.get("to"),3000));applicationPeriodStatics.add(applicationPeriodStatic);//5秒period = appPeriodstatic.get("5秒");applicationPeriodStatic = new ApplicationPeriodStatic();applicationPeriodStatic.setPeriod("5秒");applicationPeriodStatic.setDocCount(ResultUtil.longValue(period.get("doc_count"),0l));applicationPeriodStatic.setFrom(ResultUtil.intValue(period.get("from"),3000));applicationPeriodStatic.setTo(ResultUtil.intValue(period.get("to"),5000));applicationPeriodStatics.add(applicationPeriodStatic);//5秒以上period = appPeriodstatic.get("5秒以上");applicationPeriodStatic = new ApplicationPeriodStatic();applicationPeriodStatic.setPeriod("5秒以上");applicationPeriodStatic.setDocCount(ResultUtil.longValue(period.get("doc_count"),0l));applicationPeriodStatic.setFrom(ResultUtil.intValue(period.get("from"),5000));applicationPeriodStatics.add(applicationPeriodStatic);applicationStatic.setApplicationPeriodStatics(applicationPeriodStatics);applicationStatics.add(applicationStatic);}//返回統計結果return applicationStatics;}return null;} }

3.1.3 執行測試用例?

Java代碼?

@Testpublic void testAppStatic(){TraceExtraCriteria traceExtraCriteria = new TraceExtraCriteria();traceExtraCriteria.setStartTime(1516304868072l);traceExtraCriteria.setEndTime(1516349516377l);ClientInterface clientUtil = ElasticSearchHelper.getConfigRestClientUtil("esmapper/estrace/ESTracesMapper.xml");//通過下面的方法先得到查詢的json報文，然后再通過MapRestResponse查詢遍歷結果，調試的時候打開String response的注釋//String response = clientUtil.executeRequest("trace-*/_search","applicationSumStatic",traceExtraCriteria);//System.out.println(response);MapRestResponse restResponse = clientUtil.search("trace-*/_search","applicationSumStatic",traceExtraCriteria);List<Map<String,Object>> appstatics = restResponse.getAggBuckets("applicationsums",new ESTypeReference<List<Map<String,Object>>>(){});int doc_count_error_upper_bound = restResponse.getAggAttribute("applicationsums","doc_count_error_upper_bound",int.class);int sum_other_doc_count = restResponse.getAggAttribute("applicationsums","sum_other_doc_count",int.class);System.out.println("doc_count_error_upper_bound:"+doc_count_error_upper_bound);System.out.println("sum_other_doc_count:"+sum_other_doc_count);for(int i = 0; i < appstatics.size(); i ++){Map<String,Object> map = appstatics.get(i);//應用名稱String appName = (String)map.get("key");//應用總訪問量int totalsize = (int)map.get("doc_count");//獲取成功數和失敗數List<Map<String,Object>> appstatic = ResultUtil.getAggBuckets(map ,"successsums",new ESTypeReference<List<Map<String,Object>>>(){});doc_count_error_upper_bound = ResultUtil.getAggAttribute(map ,"successsums","doc_count_error_upper_bound",int.class);sum_other_doc_count = ResultUtil.getAggAttribute(map ,"successsums","sum_other_doc_count",int.class);System.out.println("doc_count_error_upper_bound:"+doc_count_error_upper_bound);System.out.println("sum_other_doc_count:"+sum_other_doc_count);/**"buckets": [{"key": 0,"doc_count": 30}]*///key 0int success = 0;//成功數int failed = 0;//失敗數for(int j = 0; j < appstatic.size(); i ++){Map<String,Object> stats = appstatic.get(i);int key = (int) stats.get("key");//成功和錯誤標識if(key == 0)success = (int)stats.get("doc_count");else if(key == 1)failed = (int)stats.get("doc_count");}}}

3.1.4 獲取元數據信息的測試方法?

java代碼?

3.2 案例2?簡單的term統計

3.2.1 定義dsl

建立dsl配置文件esmapper/testagg.xml，定義termAgg：

3.2.2 執行dsl

@Testpublic void termAgg(){ClientInterface clientInterface = ElasticSearchHelper.getConfigRestClientUtil("esmapper/testagg.xml");//ESDatas<Map> traces = clientInterface.searchAll("trace-*",1000,Map.class);//獲取總記錄集合Map params = new HashMap();//聚合統計條件參數params.put("application","testweb");SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:dd");try {params.put("startTime",format.parse("1999-01-01 00:00:00").getTime());params.put("endTime",new Date().getTime());params.put("rpc","/testweb/jsp/logoutredirect.jsp");} catch (ParseException e) {e.printStackTrace();}//一行代碼，執行每個服務的訪問量總數統計ESAggDatas<LongAggHit> response = clientInterface.searchAgg("trace-*/_search",//從trace-開頭的索引表中檢索數據"termAgg", //配置在esmapper/testagg.xml中的dsl語句params, //dsl語句termAgg中需要的查詢參數LongAggHit.class, //封裝聚合統計中每個服務地址及服務訪問量的地址"traces"); //term統計桶的名稱，參見dsl語句List<LongAggHit> aggHitList = response.getAggDatas();//每個服務的訪問量long totalSize = response.getTotalSize();//總訪問量}

3.3 案例3?簡單的cardinality統計

3.3.1 定義dsl

建立dsl配置文件esmapper/testagg.xml，定義candicateAgg：

3.3.2 執行dsl

@Testpublic void candicateAgg(){ClientInterface clientInterface = ElasticSearchHelper.getConfigRestClientUtil("esmapper/testagg.xml");Map params = null;//單值聚合統計條件參數//一行代碼，執行服務基數統計ESAggDatas<SingleLongAggHit> response = clientInterface.searchAgg("trace-*/_search","candicateAgg",params,SingleLongAggHit.class,"traces");SingleLongAggHit aggHitList = response.getSingleAggData();long value = aggHitList.getValue();long totalSize = response.getTotalSize();//總訪問量}

4.相關資料?

高性能elasticsearch ORM開發庫使用介紹?

https://my.oschina.net/bboss/blog/1556866?

bboss elasticsearch交流：166471282

轉載于:https://my.oschina.net/bboss/blog/1610016

總結

以上是生活随笔為你收集整理的Elasticsearch聚合查询案例分享的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇：初学者应该看看的6个free命令例子
下一篇： docker:Error running