當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

本地编译Hadoop2.8.0源码总结和问题解决(转自：http://blog.csdn.net/young_kim1/article/details/50324345)

發布時間：2024/9/27 编程问答 24 豆豆

生活随笔收集整理的這篇文章主要介紹了本地编译Hadoop2.8.0源码总结和问题解决(转自：http://blog.csdn.net/young_kim1/article/details/50324345) 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

編譯準備

1、下載所需的軟件

先去官網下載hadoop2.8.0源碼并解壓，打開解壓目錄下的BUILDING.txt，編譯過程和需要的軟件其實就是根據這個文檔里的描述來的。

(可以通過命令下載：wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.8.0/hadoop-2.8.0-src.tar.gz)

Requirements:* Unix System * JDK 1.7+ * Maven 3.0 or later * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac * Zlib devel (if compiling native code) * openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance ) * Jansson C XML parsing library ( if compiling libwebhdfs ) * Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs ) * Internet connection for first build (to fetch all Maven and Hadoop dependencies)

這是編譯所需要的軟件，下載戳這里。

包括：

JDK1.7+

maven 3.0 or later

findbugs 1.3.9

protocolBuffer 2.5.0

cmake 2.6

zlib-devel

openssl-devel

根據網友的資料，也需要安裝autoconf automake gcc等。

2、安裝軟件

1> 安裝JDK1.7并配置環境變量，這里就不贅述了，具體看前面的文檔。

2> 安裝各種庫

yum -y install svn ncurses-devel gcc* yum -y install lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel

3> 安裝maven

解壓安裝并配置環境變量，具體目錄結構看前面的博客。

tar zxvf apache-maven-3.3.9-bin.tar.gz vi /etc/profile

在profile文件末尾追加

export MAVEN_HOME=/home/toto/software/apache-maven-3.3.9 export MAVEN_OPTS="-Xms256m -Xmx512m" (可以不設置) export PATH=$PATH:$MAVEN_HOME/bin進入/home/toto/software/apache-maven-3.3.9/conf，修改settings.xml中臨時產生的本地maven庫的位置 <localRepository>/home/toto/software/repo</localRepository>
保存并使環境變量生效， source /etc/profile,輸入 mvn -version,有下面輸出結果則安裝并配置正確。

[root@hadoop ~]# mvn -version Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /home/toto/software/apache-maven-3.3.9 Java version: 1.8.0_73, vendor: Oracle Corporation Java home: /usr/local/java/jdk1.8.0_73/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-573.el6.x86_64", arch: "amd64", family: "unix" [root@hadoop ~]#

4> 安裝protocolBuffer

解壓安裝并配置環境變量。

tar zxvf protobuf-2.5.0.tar.gz cd /home/toto/software/protobuf-2.5.0 ./configure make make install

輸入protoc --version,有下面輸出結果則安裝并配置正確。

[hadoop@Master ~]$ protoc --version libprotoc 2.5.0 [hadoop@Master ~]$

5> 安裝findbugs

解壓安裝并配置環境變量。

unzip findbugs-1.3.9.zip vi /etc/profile

在profile文件末尾追加

export FINDBUGS_HOME=/home/toto/software/findbugs-1.3.9 export PATH=$PATH:$FINDBUGS_HOME/bin

保存并使環境變量生效，source /etc/profile,輸入findbugs -version,有下面輸出結果則安裝并配置正確。

[hadoop@Master ~]$ findbugs -version 1.3.9 [hadoop@Master ~]$

開始編譯

首先保證主機能上網（虛擬機怎么上網的點這里），在編譯過程中網絡保持暢通；進入到hadoop2.8.0源碼的解壓目錄下，輸入下面命令：

cd /home/toto/software/hadoop-2.8.0-srcmvn package -Pdist,native -DskipTests -Dtar

或者這個

cd /home/toto/software/hadoop-2.8.0-srcmvn package -Pdist,native,docs,src -DskipTests -Dtar

前面只編譯本地代碼，后者編譯本地代碼和文檔，因此前者速度較快。

接下來就是漫長的等待，等出現這個就說明編譯成功。

[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 3.533 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 2.023 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.679 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.275 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.875 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.856 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.340 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.534 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 5.398 s] [INFO] Apache Hadoop Common ............................... SUCCESS [03:02 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 11.653 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 24.501 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.112 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [07:28 min] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 41.608 s] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 10.673 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 7.225 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.057 s] [INFO] hadoop-yarn ........................................ SUCCESS [ 0.110 s] [INFO] hadoop-yarn-api .................................... SUCCESS [03:36 min] [INFO] hadoop-yarn-common ................................. SUCCESS [ 45.418 s] [INFO] hadoop-yarn-server ................................. SUCCESS [ 0.164 s] [INFO] hadoop-yarn-server-common .......................... SUCCESS [ 12.942 s] [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 19.200 s] [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 3.315 s] [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 7.855 s] [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 24.347 s] [INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 6.439 s] [INFO] hadoop-yarn-client ................................. SUCCESS [ 6.393 s] [INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 3.445 s] [INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.075 s] [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.304 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 2.026 s] [INFO] hadoop-yarn-site ................................... SUCCESS [ 0.155 s] [INFO] hadoop-yarn-registry ............................... SUCCESS [ 7.255 s] [INFO] hadoop-yarn-project ................................ SUCCESS [ 11.871 s] [INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.254 s] [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 26.029 s] [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 25.002 s] [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 3.792 s] [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 7.797 s] [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 5.143 s] [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 6.771 s] [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.837 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.513 s] [INFO] hadoop-mapreduce ................................... SUCCESS [ 6.842 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.355 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 14.910 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 2.844 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 6.931 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.937 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 2.499 s] [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 2.268 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.739 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 5.793 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 4.444 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 4.258 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 47.689 s] [INFO] Apache Hadoop Client ............................... SUCCESS [ 19.524 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.305 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 5.581 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 25.708 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.281 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:53 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 24:49 min [INFO] Finished at: 2015-12-11T20:29:45+08:00 [INFO] Final Memory: 110M/493M [INFO] ------------------------------------------------------------------------

編譯好的文件在../hadoop-dist/target/hadoop-2.8.0.tar.gz下。

最后編譯后出的效果：

cd /home/toto/software/hadoop-2.8.0-src/hadoop-dist/target

編譯中遇到的問題

錯誤1

Connection to?http://repo.maven.apache.org?refused

表示連接maven遠程倉庫拒絕，此時再運行一下編譯命令，就會接著下載jar包。

錯誤2

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-nfs: Compilation failure: Compilation failure: [ERROR] /home/hadoop/toolkits/hadoop-2.7.1-src/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/XDR.java:[23,30] package org.jboss.netty.buffer does not exist

這個錯誤估計很少遇到，這是因為我嫌repo.maven.apache.org這個網站比較慢，更改的第三方鏡像，導致maven中的settings文件配置錯誤，后來采用默認的就行，雖然慢點。

錯誤3

[ERROR] around Ant part ...<exec dir="/opt/soft/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target" executable="sh" failonerror="true">... @ 10:123 in /opt/soft/hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml [ERROR] -> [Help 1]

這是由于tomcat的apache-tomcat-6.0.41.tar.gz包太大，沒有下載完整，可以到.../hadoop-2.7.1-src/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads/apache-tomcat-6.0.41.tar.gz這個目錄下，刪除重新下載。

提醒：?
1、有時候編譯過程中會出現下載某個包的時間太久，這是由于連接網站的過程中會出現假死，此時按ctrl+c，重新運行編譯命令。?
2、如果出現缺少了某個文件的情況，則要先清理maven(使用命令?mvn clean) 再重新編譯。

====================================

寫在最后：

經過了較長時間的折磨，終于成功，可能還是自己比較菜吧，有編譯不成功的歡迎留言探討。

總結

以上是生活随笔為你收集整理的本地编译Hadoop2.8.0源码总结和问题解决(转自：http://blog.csdn.net/young_kim1/article/details/50324345)的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。