【hive】——metastore的三种模式
生活随笔
收集整理的這篇文章主要介紹了
【hive】——metastore的三种模式
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
Hive中metastore（元數據存儲）的三種方式：
- 內嵌Derby方式
- Local方式
- Remote方式
[一]、內嵌Derby方式
這個是Hive默認的啟動模式，一般用于單元測試，這種存儲方式有一個缺點：在同一時間只能有一個進程連接使用數據庫。
hive-site.xml?中jdbc URL、驅動、用戶名、密碼等的配置信息如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> ??<name>javax.jdo.option.ConnectionURL</name> ??<value>jdbc:derby:;databaseName=metastore_db;create=true</value> ??<description>JDBC connect string for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionDriverName</name> ??<value>org.apache.derby.jdbc.EmbeddedDriver</value> ??<description>Driver class name for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionUserName</name> ??<value>APP</value> ??<description>username to use against metastore database</description> </property> <property> ??<name>javax.jdo.option.ConnectionPassword</name> ??<value>mine</value> ??<description>password to use against metastore database</description> </property> <property> ??<name>hive.metastore.warehouse.dir</name> ??<value>file:///Users/micmiu/tmp/hive/warehouse</value> ??<description>unit test data goes in here on your local filesystem</description> </property> <!-- micmiu.com --> |
執行初始化命令：schematool -dbType derby -initSchema
查看初始化后的信息：?schematool -dbType derby -info
配置完成后就可在shell中以CLI的方式訪問hive 進行操作驗證。
[二]、Local方式
以本地Mysql數據庫為例：創建好用戶：hive；database：hive。
配置文件 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> ??<name>javax.jdo.option.ConnectionURL</name> ??<value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value> ??<description>JDBC connect string for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionDriverName</name> ??<value>com.mysql.jdbc.Driver</value> ??<description>Driver class name for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionUserName</name> ??<value>hive</value> ??<description>username to use against metastore database</description> </property> <property> ??<name>javax.jdo.option.ConnectionPassword</name> ??<value>micmiu</value> ??<description>password to use against metastore database</description> </property> <property> ??<name>hive.metastore.warehouse.dir</name> ??<!-- base hdfs path --> ??<value>/user/hive/warehouse</value> ??<description>location of default database for the warehouse</description> </property> <!-- micmiu.com --> |
ps：需要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中
如果是第一次需要執行初始化命令：schematool -dbType mysql -initSchema
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | micmiu-mbp:mysql micmiu$ schematool -dbType mysql -initSchema 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/02/17 14:46:08 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative Metastore connection URL: jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive Starting metastore schema initialization to 0.12.0 Initialization script hive-schema-0.12.0.mysql.sql Initialization script completed schemaTool completeted |
查看初始化后信息?schematool -dbType mysql -info
初始化后查看mysql中表情況：show tables;
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | mysql> show tables; +---------------------------+ | Tables_in_hive????????????| +---------------------------+ | BUCKETING_COLS????????????| | CDS?????????????????????? | | COLUMNS_V2????????????????| | DATABASE_PARAMS?????????? | | DBS?????????????????????? | | DB_PRIVS??????????????????| | DELEGATION_TOKENS???????? | | GLOBAL_PRIVS??????????????| | IDXS??????????????????????| | INDEX_PARAMS??????????????| | MASTER_KEYS?????????????? | | NUCLEUS_TABLES????????????| | PARTITIONS????????????????| | PARTITION_EVENTS??????????| | PARTITION_KEYS????????????| | PARTITION_KEY_VALS????????| | PARTITION_PARAMS??????????| | PART_COL_PRIVS????????????| | PART_COL_STATS????????????| | PART_PRIVS????????????????| | ROLES???????????????????? | | ROLE_MAP??????????????????| | SDS?????????????????????? | | SD_PARAMS???????????????? | | SEQUENCE_TABLE????????????| | SERDES????????????????????| | SERDE_PARAMS??????????????| | SKEWED_COL_NAMES??????????| | SKEWED_COL_VALUE_LOC_MAP??| | SKEWED_STRING_LIST????????| | SKEWED_STRING_LIST_VALUES | | SKEWED_VALUES???????????? | | SORT_COLS???????????????? | | TABLE_PARAMS??????????????| | TAB_COL_STATS???????????? | | TBLS??????????????????????| | TBL_COL_PRIVS???????????? | | TBL_PRIVS???????????????? | | TYPES???????????????????? | | TYPE_FIELDS?????????????? | | VERSION?????????????????? | +---------------------------+ 41 rows in set (0.00 sec) |
配置完成后就可在shell中以CLI的方式訪問hive 進行操作驗證。
[三]、Remote方式
以Mysql數據庫(192.168.6.77)為例：創建好用戶：hive；database：hive_meta。Remote方式需要分別配置服務端和客戶端的配置文件：
服務端的 hive-site.xml 中jdbc URL、驅動、用戶名、密碼等屬性值配置如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | <property> ??<name>javax.jdo.option.ConnectionURL</name> ??<value>jdbc:mysql://192.168.6.77/hive_meta?createDatabaseIfNotExist=true</value> ??<description>JDBC connect string for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionDriverName</name> ??<value>com.mysql.jdbc.Driver</value> ??<description>Driver class name for a JDBC metastore</description> </property> <property> ??<name>javax.jdo.option.ConnectionUserName</name> ??<value>hive</value> ??<description>username to use against metastore database</description> </property> <property> ??<name>javax.jdo.option.ConnectionPassword</name> ??<value>micmiu</value> ??<description>password to use against metastore database</description> </property> <property> ??<name>hive.metastore.warehouse.dir</name> ??<!-- base hdfs path --> ??<value>/user/hive/warehouse</value> ??<description>base hdfs path :location of default database for the warehouse</description> </property> <!-- micmiu.com --> |
ps：需要把mysql的驅動包copy到目錄 <HIVE_HOME>/lib 中
如果是第一次需要執行初始化命令：schematool -dbType mysql -initSchema
客戶端中配置內容修改如下：
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | <!-- thrift://<host_name>:<port> 默認端口是9083 --> <property> ??<name>hive.metastore.uris</name> ??<value>thrift://192.168.6.77:9083</value> ??<description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description> </property> <!--??hive表的默認存儲路徑 --> <property> ??<name>hive.metastore.warehouse.dir</name> ??<value>/user/hive/warehouse</value> ??<description>location of default database for the warehouse</description> </property> |
hive metastore?服務端啟動命令：
hive --service metastore -p <port_num>
如果不加端口默認啟動：hive --service metastore，則默認監聽端口是：9083?，注意客戶端中的端口配置需要和啟動監聽的端口一致。服務端啟動正常后，客戶端就可以執行hive操作了。
參考：
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin
轉載于:https://www.cnblogs.com/zhengrunjian/p/4546032.html
總結
以上是生活随笔為你收集整理的【hive】——metastore的三种模式的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 2021年第十八届五一数学建模竞赛题目
- 下一篇： Fiddler 抓包详细使用教程