生活随笔
收集整理的這篇文章主要介紹了
hadoop-2.5.1 源码编译
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
hadoop-2.5.1 編譯步驟 ( 只需一個節點即可 )
環境:
系統?? CentOS 6.4 64 位??
m1????? 172.103.55.12 軟件:? jdk1.7、hadoop- 2.5.1- src.tar.gz、maven3.2.3、 protobuf2.5.0
注:需要連接互聯網 ?
1. ?????? ? 安裝 配置 JDK (略)
2. ??????? 安裝基本應用程序 ?( 需要連接互聯網網 )?
yum -y install??svn? ?ncurses-devel? ?gcc*? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ? yum -y install lzo-devel zlib-devel autoconf? ? automake? ? libtool? ? cmake? ???openssl –devel
3. ?????? 安裝 ?maven
下載 apache-maven-3.2.3-bin.tar.gz? ? 并解壓 tar? ?zxvf? ?apache- maven- 3.2.3- bin.tar.gz
配置環境變量 export MAVEN_HOME=/usr/local/program/maven/ apache- maven- 3.2.3 export PATH=$PATH:$MAVEN_HOME/bin? ?? ?? 添加之后執行sourc?/etc/profile
4. ?????安裝 ?protobuf (不安裝,編譯將無法完成)
hadoop 使用 protocol buffer 進行通信,需要下載和安裝 protobhf-2.5.0.tar.gz ;由于官網已經無法下載了,
可以到百度云盤下載 http://pan.baidu.com/s/1pJlZubT
編譯安裝 ?protobuf ① ??cd? ?protobuf - 2.5.0? ? ② ??./configure? ③ ??make ④ ??make install
檢測 ??protoc –version
?
5. ?????編譯 ?hadoop-2.5.1.src
a) ?????下載 ??hadoop-2.5.1-src.tat.gz ? 源碼包下載地址 ?? http://apache.claz.org/hadoop/common/hadoop-2.5.1/ ? 紅色標志 ??hadoop-2.5.1- src.tar.gz? 為源碼包 ?
b) ????解壓 hadoop-2.5.1-src.tar.gz
?
c) ?????開始編譯
進入hadoop-2.5.1.src 目錄:
執行命令 ? ?mvn clean package –Pdist,native –DskipTests –Dtar?
(或者 mvn? package? -Pdist,native? -DskipTests -Dtar )?
???? 此步驟要求逐級能訪問公網,并且此步驟比較漫長,請耐心等待。
?
經過漫長的等待后,終于看到了Building SUCCESS:
[INFO] Reactor Summary: [INFO]? [INFO] Apache Hadoop Main ................................. SUCCESS [ ?4.389 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ ?3.163 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ ?6.377 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ ?0.562 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ ?3.136 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ ?6.757 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ ?5.506 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ ?6.269 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ ?4.341 s] [INFO] Apache Hadoop Common ............................... SUCCESS [03:25 min] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 23.998 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ ?0.160 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [07:52 min] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [11:03 min] [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [03:23 min] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 14.324 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ ?0.343 s] [INFO] hadoop-yarn ........................................ SUCCESS [ ?0.092 s] [INFO] hadoop-yarn-api .................................... SUCCESS [02:46 min] [INFO] hadoop-yarn-common ................................. SUCCESS [02:37 min] [INFO] hadoop-yarn-server ................................. SUCCESS [ ?1.142 s] [INFO] hadoop-yarn-server-common .......................... SUCCESS [01:02 min] [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [02:26 min] [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 22.258 s] [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 16.444 s] [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 51.768 s] [INFO] hadoop-yarn-server-tests ........................... SUCCESS [ ?9.864 s] [INFO] hadoop-yarn-client ................................. SUCCESS [ 16.057 s] [INFO] hadoop-yarn-applications ........................... SUCCESS [ ?0.162 s] [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ ?7.307 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ ?3.243 s] [INFO] hadoop-yarn-site ................................... SUCCESS [ ?0.099 s] [INFO] hadoop-yarn-project ................................ SUCCESS [ 18.997 s] [INFO] hadoop-mapreduce-client ............................ SUCCESS [ ?2.306 s] [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [01:34 min] [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [01:05 min] [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 20.109 s] [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 22.691 s] [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 30.368 s] [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:30 min] [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 19.958 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 24.097 s] [INFO] hadoop-mapreduce ................................... SUCCESS [ 17.757 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [01:59 min] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 48.464 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 14.187 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 15.098 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 13.630 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ ?8.080 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ ?6.496 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 23.855 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 17.886 s] [INFO] Apache Hadoop Client ............................... SUCCESS [ 18.728 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ ?1.060 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 35.296 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.476 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ ?0.134 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:20 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 51:55 min [INFO] Finished at: 2014-12-09T19:16:49-08:00 [INFO] Final Memory: 142M/350M [INFO] ------------------------------------------------------------------------
編譯成功后,??/hadoop-dist/target/hadoop-2.5.1.tar.gz? ?就是我們需要的文件了;
?可能遇到的問題:
報錯[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space
分配內存不足,參考如下為maven配置JVM參數:?export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
我直接在profile文件中:export MAVEN_OPTS=-Xmx1024
創作挑戰賽 新人創作獎勵來咯,堅持創作打卡瓜分現金大獎
總結
以上是生活随笔 為你收集整理的hadoop-2.5.1 源码编译 的全部內容,希望文章能夠幫你解決所遇到的問題。
如果覺得生活随笔 網站內容還不錯,歡迎將生活随笔 推薦給好友。