hive的啟動(dòng)需要使用到zookeeper, 所以, 要么自己搭建zookeeper, 要么跟其它東西一起使用, 我這里做的是跟hbase一起使用的zookeeper, 因?yàn)閔base自帶zookeeper, hbase啟動(dòng)就會(huì)啟動(dòng)zookeeper, 而hive默認(rèn)會(huì)連接本機(jī)的2181端口, 所以我這里選擇在slaver3上使用hive.


集群的搭建以及機(jī)器的分配見(jiàn)hadoop搭建: http://phey.cc/multinode_hadoop20.html
以及hbase集群搭建http://phey.cc/Install_hbase_cluster.html


解壓hive包后拷貝環(huán)境變量模板到指定文件
[cc@slaver3 ~]$ cp hive-0.12.0-cdh5.0.1/conf/hive-env.sh.template hive-0.12.0-cdh5.0.1/conf/hive-env.sh
[cc@slaver3 ~]$ ▊




編輯環(huán)境變量， 一個(gè)是hadoop的安裝目錄，一個(gè)是hbase的jar位置，如果hbase和hive的jar版本不對(duì)會(huì)報(bào)錯(cuò)
[cc@slaver3 ~]$ vim hive-0.12.0-cdh5.0.1/conf/hive-env.sh
export HADOOP_HOME=/home/cc/hadoop-2.3.0-cdh5.0.0
export HIVE_AUX_JARS_PATH=/home/cc/hbase-0.96.1.1-cdh5.0.1/lib
[cc@slaver3 ~]$ ▊




啟動(dòng)hive，指定hbase的RPC端口
[cc@slaver3 hive-0.12.0-cdh5.0.1]$ bin/hive -hiveconf hbase.master=master1:60000
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/08/21 21:34:07 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative


Logging initialized using configuration in jar:file:/home/cc/hive-0.12.0-cdh5.0.1/lib/hive-common-0.12.0-cdh5.0.1.jar!/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/cc/hadoop-2.3.0-cdh5.0.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/cc/hive-0.12.0-cdh5.0.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/cc/hbase-0.96.1.1-cdh5.0.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hive> ▊




在hive中建表。在hive新建一個(gè)名為cctabl的表，這個(gè)表映射到hbase的表名是cc，cctable表里面的int類(lèi)型的key對(duì)應(yīng)了cc表里面的row key，cctable里面類(lèi)型是string的value對(duì)應(yīng)了cc表里面的cf:val
hive> CREATE TABLE cctable (key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:val") TBLPROPERTIES ("hbase.table.name" = "cc");
OK
Time taken: 9.302 seconds
hive> ▊




在hbase中可以看到結(jié)果, 創(chuàng)建了一個(gè)表
hbase(main):011:0> list
TABLE ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
cc ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
1 row(s) in 0.0250 seconds


=> ["cc"]
hbase(main):012:0> describe 'cc'
DESCRIPTION ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ENABLED ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
?'cc', {NAME => 'cf', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIO true ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
?NS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
? BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??
1 row(s) in 0.0390 seconds


hbase(main):013:0> ▊




如果不希望hive去創(chuàng)建表而是使用hbase已經(jīng)有的表, 那么創(chuàng)建表的時(shí)候加上external參數(shù)就可以了, 例如
hive> CREATE EXTERNAL TABLE cctable (key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:val") TBLPROPERTIES ("hbase.table.name" = "cc");
OK
Time taken: 9.302 seconds
hive> ▊




由于有那個(gè)映射關(guān)系，所以往hbase里面插入數(shù)據(jù)可以在hive里面查詢(xún)，不過(guò)hive不支持類(lèi)似于mysql的insert語(yǔ)句，所以方便的也只能往hbase里面插入數(shù)據(jù)


向hbase里面插入數(shù)據(jù)
hbase(main):013:0> put 'cc', '1', 'cf:val', 'hello cc!'
0 row(s) in 0.0120 seconds


hbase(main):014:0> ▊




在hive里面查詢(xún)
hive> select * from cctable;
OK
1 ? ? ? hello cc!
Time taken: 30.838 seconds, Fetched: 1 row(s)
hive> ▊




雖然hbase不支持count方法去計(jì)算行數(shù)，但是hive可以，不過(guò)這個(gè)會(huì)被轉(zhuǎn)換成mapreduce過(guò)程，去執(zhí)行
hive> select count(*) from cctable;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
? set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
? set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
? set mapred.reduce.tasks=<number>
Starting Job = job_1408532552242_0005, Tracking URL = http://master1:8088/proxy/application_1408532552242_0005/
Kill Command = /home/cc/hadoop-2.3.0-cdh5.0.0/bin/hadoop job ?-kill job_1408532552242_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2014-08-22 10:15:00,861 Stage-1 map = 0%, ?reduce = 0%
2014-08-22 10:15:14,479 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:15,525 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:16,572 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:17,617 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:18,662 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:19,707 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:20,753 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:21,798 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:22,842 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:23,889 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:24,937 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:25,991 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:27,040 Stage-1 map = 100%, ?reduce = 0%, Cumulative CPU 2.73 sec
2014-08-22 10:15:28,094 Stage-1 map = 100%, ?reduce = 100%, Cumulative CPU 4.89 sec
2014-08-22 10:15:29,143 Stage-1 map = 100%, ?reduce = 100%, Cumulative CPU 4.89 sec
MapReduce Total cumulative CPU time: 4 seconds 890 msec
Ended Job = job_1408532552242_0005
MapReduce Jobs Launched:?
Job 0: Map: 1 ?Reduce: 1 ? Cumulative CPU: 4.89 sec ? HDFS Read: 236 HDFS Write: 2 SUCCESS
Total MapReduce CPU Time Spent: 4 seconds 890 msec
OK
1
Time taken: 77.571 seconds, Fetched: 1 row(s)
hive> ▊

轉(zhuǎn)載于:https://www.cnblogs.com/jamesf/p/4751467.html

總結(jié)

以上是生活随笔為你收集整理的Hive与Hbase结合使用的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。