當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Hive分析窗口函数(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

發布時間：2023/12/10 编程问答 30 豆豆

生活随笔收集整理的這篇文章主要介紹了 Hive分析窗口函数(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

GROUPING SETS

該關鍵字可以實現同一數據集的多重group by操作。事實上GROUPING SETS是多個GROUP BY進行UNION ALL操作的簡單表達，它僅僅使用一個stage完成這些操作。GROUPING SETS的子句中如果包含()數據集，則表示整體聚合。

Aggregate Query with GROUPING SETS

Equivalent Aggregate Query with GROUP BY

SELECT a, b, SUM( c ) FROM tab1 GROUP BY a, b GROUPING SETS ( (a, b), a, b, ( ) )	SELECT a, b, SUM( c ) FROM tab1 GROUP BY a, b UNION SELECT a, null, SUM( c ) FROM tab1 GROUP BY a, null UNION SELECT null, b, SUM( c ) FROM tab1 GROUP BY null, b UNION SELECT null, null, SUM( c ) FROM tab1
SELECT a, b, SUM( c ) FROM tab1 GROUP BY a, b GROUPING SETS ( (a,b), a)	SELECT a, b, SUM( c ) FROM tab1 GROUP BY a, b UNION SELECT a, null, SUM( c ) FROM tab1 GROUP BY a
SELECT a, b, SUM(c) FROM tab1 GROUP BY a, b GROUPING SETS ( (a,b) )	SELECT a, b, SUM(c) FROM tab1 GROUP BY a, b
SELECT a,b, SUM( c ) FROM tab1 GROUP BY a, b GROUPING SETS (a,b)	SELECT a, null, SUM( c ) FROM tab1 GROUP BY a UNION SELECT null, b, SUM( c ) FROM tab1 GROUP BY b

ROLLUP

擴展了GROUTING SETS。

其中count(d) 可以換成其他聚合函數例如：sum(d)

select a, b, c, count(d) from table group by a, b, c WITH ROLLUP; // 等價于下面語句 select a, b, c from table group by a, b, c GROUPING SETS((a,b,c),(a,b),(a),());

CUBE

擴展了GROUTING SETS，對各種條件進行聚合。

其中count(d) 可以換成其他聚合函數例如：sum(d)

select a, b, c,count(d) from table group by a, b, c WITH ROLLUP; // 等價于下面語句 select a, b, c from table group by a, b, c GROUPING SETS((a,b,c),(a,b),(a,c),(b,c),(a),(b),(c),());

聚合條件 HAVING

having用于在組內進行過濾。

select cid,max(price) mx from orders group by cid having mx > 1000; //等價于下面的子查詢語句 select t.cid, t.mx from (select cid, max(price) mx from orders group by cid) t where t.mx > 1000;

Cubes and Rollups

The general syntax is WITH CUBE/ROLLUP. It is used with the GROUP BY only. CUBE creates a subtotal of all possible combinations of the set of column in its argument. Once we compute a CUBE on a set of dimension, we can get answer to all possible aggregation questions on those dimensions.It might be also worth mentioning here that? GROUP BY a, b, c WITH CUBE is equivalent to? GROUP BY a, b, c GROUPING SETS ( (a, b, c), (a, b), (b, c), (a, c), (a), (b), (c), ( )).ROLLUP clause is used with GROUP BY to compute the aggregate at the hierarchy levels of a dimension. GROUP BY a, b, c with ROLLUP assumes that the hierarchy is "a" drilling down to "b" drilling down to "c".GROUP BY a, b, c, WITH ROLLUP is equivalent to GROUP BY a, b, c GROUPING SETS ( (a, b, c), (a, b), (a), ( )).

實例：

轉載地址：

??Hive分析窗口函數(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

這幾個分析函數通常用于OLAP中，不能累加，而且需要根據不同維度上鉆和下鉆的指標統計，比如，分小時、天、月的UV數。

Hive版本為 apache-hive-0.13.1

數據準備：

2015-03,2015-03-10,cookie12015-03,2015-03-10,cookie52015-03,2015-03-12,cookie72015-04,2015-04-12,cookie32015-04,2015-04-13,cookie22015-04,2015-04-13,cookie42015-04,2015-04-16,cookie42015-03,2015-03-10,cookie22015-03,2015-03-10,cookie32015-04,2015-04-12,cookie52015-04,2015-04-13,cookie62015-04,2015-04-15,cookie32015-04,2015-04-15,cookie22015-04,2015-04-16,cookie1CREATE EXTERNAL TABLE lxw1234 (month STRING,day STRING, cookieid STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' stored as textfile location '/tmp/lxw11/';hive> select * from lxw1234;OK2015-03 2015-03-10 cookie12015-03 2015-03-10 cookie52015-03 2015-03-12 cookie72015-04 2015-04-12 cookie32015-04 2015-04-13 cookie22015-04 2015-04-13 cookie42015-04 2015-04-16 cookie42015-03 2015-03-10 cookie22015-03 2015-03-10 cookie32015-04 2015-04-12 cookie52015-04 2015-04-13 cookie62015-04 2015-04-15 cookie32015-04 2015-04-15 cookie22015-04 2015-04-16 cookie1

GROUPING SETS

在一個GROUP BY查詢中，根據不同的維度組合進行聚合，等價于將不同維度的GROUP BY結果集進行UNION ALL

SELECT month,day,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID FROM lxw1234 GROUP BY month,day GROUPING SETS (month,day) ORDER BY GROUPING__ID;month day uv GROUPING__ID------------------------------------------------2015-03 NULL 5 12015-04 NULL 6 1NULL 2015-03-10 4 2NULL 2015-03-12 1 2NULL 2015-04-12 2 2NULL 2015-04-13 3 2NULL 2015-04-15 2 2NULL 2015-04-16 2 2等價于 SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month UNION ALL SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY day

再如：

SELECT month,day,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID FROM lxw1234 GROUP BY month,day GROUPING SETS (month,day,(month,day)) ORDER BY GROUPING__ID;month day uv GROUPING__ID------------------------------------------------2015-03 NULL 5 12015-04 NULL 6 1NULL 2015-03-10 4 2NULL 2015-03-12 1 2NULL 2015-04-12 2 2NULL 2015-04-13 3 2NULL 2015-04-15 2 2NULL 2015-04-16 2 22015-03 2015-03-10 4 32015-03 2015-03-12 1 32015-04 2015-04-12 2 32015-04 2015-04-13 3 32015-04 2015-04-15 2 32015-04 2015-04-16 2 3等價于SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month UNION ALL SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY dayUNION ALL SELECT month,day,COUNT(DISTINCT cookieid) AS uv,3 AS GROUPING__ID FROM lxw1234 GROUP BY month,day

其中的?GROUPING__ID，表示結果屬于哪一個分組集合。

CUBE

根據GROUP BY的維度的所有組合進行聚合。

SELECT month,day,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID FROM lxw1234 GROUP BY month,day WITH CUBE ORDER BY GROUPING__ID;month day uv GROUPING__ID--------------------------------------------NULL NULL 7 02015-03 NULL 5 12015-04 NULL 6 1NULL 2015-04-12 2 2NULL 2015-04-13 3 2NULL 2015-04-15 2 2NULL 2015-04-16 2 2NULL 2015-03-10 4 2NULL 2015-03-12 1 22015-03 2015-03-10 4 32015-03 2015-03-12 1 32015-04 2015-04-16 2 32015-04 2015-04-12 2 32015-04 2015-04-13 3 32015-04 2015-04-15 2 3等價于SELECT NULL,NULL,COUNT(DISTINCT cookieid) AS uv,0 AS GROUPING__ID FROM lxw1234UNION ALL SELECT month,NULL,COUNT(DISTINCT cookieid) AS uv,1 AS GROUPING__ID FROM lxw1234 GROUP BY month UNION ALL SELECT NULL,day,COUNT(DISTINCT cookieid) AS uv,2 AS GROUPING__ID FROM lxw1234 GROUP BY dayUNION ALL SELECT month,day,COUNT(DISTINCT cookieid) AS uv,3 AS GROUPING__ID FROM lxw1234 GROUP BY month,day

ROLLUP

是CUBE的子集，以最左側的維度為主，從該維度進行層級聚合。

比如，以month維度進行層級聚合：SELECT month,day,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID FROM lxw1234 GROUP BY month,dayWITH ROLLUP ORDER BY GROUPING__ID;month day uv GROUPING__ID---------------------------------------------------NULL NULL 7 02015-03 NULL 5 12015-04 NULL 6 12015-03 2015-03-10 4 32015-03 2015-03-12 1 32015-04 2015-04-12 2 32015-04 2015-04-13 3 32015-04 2015-04-15 2 32015-04 2015-04-16 2 3可以實現這樣的上鉆過程：月天的UV->月的UV->總UV --把month和day調換順序，則以day維度進行層級聚合：SELECT day,month,COUNT(DISTINCT cookieid) AS uv,GROUPING__ID FROM lxw1234 GROUP BY day,month WITH ROLLUP ORDER BY GROUPING__ID;day month uv GROUPING__ID-------------------------------------------------------NULL NULL 7 02015-04-13 NULL 3 12015-03-12 NULL 1 12015-04-15 NULL 2 12015-03-10 NULL 4 12015-04-16 NULL 2 12015-04-12 NULL 2 12015-04-12 2015-04 2 32015-03-10 2015-03 4 32015-03-12 2015-03 1 32015-04-13 2015-04 3 32015-04-15 2015-04 2 32015-04-16 2015-04 2 3可以實現這樣的上鉆過程：天月的UV->天的UV->總UV（這里，根據天和月進行聚合，和根據天聚合結果一樣，因為有父子關系，如果是其他維度組合的話，就會不一樣）

Grouping_ID函數

當我們沒有統計某一列時，它的值顯示為null，這可能與列本身就有null值沖突，這就需要一種方法區分是沒有統計還是值本來就是null。（寫一個排列組合的算法，就馬上理解了，grouping_id其實就是所統計各列二進制和）

直接拿官方文檔一個例子，O(∩_∩)O哈哈~

Column1 (key)Column2 (value)

1	NULL
1	1
2	2
3	3
3	NULL
4	5

hql統計：

SELECT key, value, GROUPING__ID, count(*) from T1 GROUP BY key, value WITH ROLLUP

統計結果如下：

????

NULL	NULL	0 ? ? 00	6
1	NULL	1 ? ? 10	2
1	NULL	3 ? ? 11	1
1	1	3 ? ? 11	1
2	NULL	1 ? ? 10	1
2	2	3 ? ? 11	1
3	NULL	1 ? ? 10	2
3	NULL	3 ? ? 11	1
3	3	3 ? ? 11	1
4	NULL	1 ? ? 10	1
4	5	3 ? ? 11	1

GROUPING__ID轉變為二進制，如果對應位上有值為null，說明這列本身值就是null。（通過類DataFilterNull.py?掃描，可以篩選過濾掉列中null、“”統計結果),

總結

cube的分組組合最全，是各個維度值的笛卡爾（包含null）組合，

rollup的各維度組合應滿足，前一維度為null后一位維度必須為null，前一維度取非null時，下一維度隨意，

grouping sets則為自定義維度，根據需要分組即可。

ps:通過grouping sets的使用可以簡化SQL，比group by單維度進行union性能更好。

這種函數，需要結合實際場景和數據去使用和研究，只看說明的話，很難理解。

官網的介紹： https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup

轉發:https://www.2cto.com/database/201708/671294.html

轉發：https://blog.csdn.net/zhoudetiankong/article/details/52527142

參考:https://blog.csdn.net/suiyingli39/article/details/53540861

參考:https://blog.csdn.net/moon_yang_bj/article/details/17200367

依據上面兩篇博客以及官網，整理

總結

以上是生活随笔為你收集整理的Hive分析窗口函数(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： android xml设置roboto字
下一篇：【Excel技巧必知必会】