背景

項(xiàng)目中需要通過一些自定義的組件來(lái)操控hive的元數(shù)據(jù)，于是使用了remote方式來(lái)存儲(chǔ)hive元數(shù)據(jù)，使用一個(gè)服務(wù)后臺(tái)作為gateway，由它來(lái)控制hive元數(shù)據(jù)。

現(xiàn)象

在windows上連接hive metastore的時(shí)候，無(wú)端的會(huì)報(bào)NullPointerException，非常費(fèi)解。

分析

看了代碼后發(fā)現(xiàn)，連接后會(huì)獲取本地用戶所在的用戶組信息（org.apache.hadoop.hive.metastore.HiveMetaStoreClient中的open方法）：

          if (isConnected && !useSasl && conf.getBoolVar(ConfVars.METASTORE_EXECUTE_SET_UGI)){// Call set_ugi, only in unsecure mode.try {UserGroupInformation ugi = Utils.getUGI();client.set_ugi(ugi.getUserName(), Arrays.asList(ugi.getGroupNames()));} catch (LoginException e) {LOG.warn("Failed to do login. set_ugi() is not successful, " +"Continuing without it.", e);} catch (IOException e) {LOG.warn("Failed to find ugi of client set_ugi() is not successful, " +"Continuing without it.", e);} catch (TException e) {LOG.warn("set_ugi() not successful, Likely cause: new client talking to old server. "+ "Continuing without it.", e);}}

ugi.getGroupNames()會(huì)去調(diào)用本地命令在windows平臺(tái)上會(huì)使用一個(gè)叫winutils的工具，但是作為客戶端開發(fā)的話不會(huì)在windows端安裝這些二進(jìn)制文件，所以代碼流程就出錯(cuò)了

  /*** a Unix command to get a given user's groups list.* If the OS is not WINDOWS, the command will get the user's primary group* first and finally get the groups list which includes the primary group.* i.e. the user's primary group will be included twice.*/public static String[] getGroupsForUserCommand(final String user) {//'groups username' command return is non-consistent across different unixesreturn (WINDOWS)? new String[] { WINUTILS, "groups", "-F", "\"" + user + "\""}: new String [] {"bash", "-c", "id -gn " + user+ "&& id -Gn " + user};

WINUTILS的初始化在如下函數(shù)中，如果path中找不到的話會(huì)返回null

  /** a Windows utility to emulate Unix commands */public static final String WINUTILS = getWinUtilsPath();public static final String getWinUtilsPath() {String winUtilsPath = null;try {if (WINDOWS) {winUtilsPath = getQualifiedBinPath("winutils.exe");}} catch (IOException ioe) {LOG.error("Failed to locate the winutils binary in the hadoop binary path",ioe);}return winUtilsPath;}

在java.lang.ProcessBuilder.java中的start中有如下判斷：

public Process start() throws IOException {// Must convert to array first -- a malicious user-supplied// list might try to circumvent the security check.String[] cmdarray = command.toArray(new String[command.size()]);cmdarray = cmdarray.clone();for (String arg : cmdarray)if (arg == null)throw new NullPointerException();// Throws IndexOutOfBoundsException if command is emptyString prog = cmdarray[0];

由于cmdarray中的第一個(gè)元素就是null，所以馬上甩出NullPointerException

toString() 中的null值檢測(cè)

另外在org.apache.hadoop.util.Shell中

ShellCommandExecutor

這個(gè)類中存在一個(gè)問題，就是toString方面沒有對(duì)成員為null的情況進(jìn)行判斷如：

    /*** Returns the commands of this instance.* Arguments with spaces in are presented with quotes round; other* arguments are presented raw** @return a string representation of the object.*/@Overridepublic String toString() {StringBuilder builder = new StringBuilder();String[] args = getExecString();for (String s : args) {if (s.indexOf(' ') >= 0) {builder.append('"').append(s).append('"');} else {builder.append(s);}builder.append(' ');}return builder.toString();}

即假如我們的命令args中有元素是null，那么這個(gè)toString也會(huì)拋出NullPointerException，因?yàn)樵跊]有判斷的情況下直接引用了對(duì)象方法（s.indexOf），記得這個(gè)問題似乎在Effective Java里看到過。一般并不會(huì)觸發(fā)這問題，可是在打開調(diào)試器的時(shí)候，它會(huì)去執(zhí)行當(dāng)前環(huán)境里對(duì)象的toString方法。所以每當(dāng)debug到相關(guān)代碼段時(shí)，總是莫名其妙的就突然爆出個(gè)NullPointerException，著實(shí)費(fèi)解了一些時(shí)間。

?

轉(zhuǎn)載于:https://www.cnblogs.com/lailailai/p/4553031.html

總結(jié)

以上是生活随笔為你收集整理的Hive Metastore 连接报错的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。