配置hadoop

1:下載hadoop-1.2.1.tar.gz

? ?在/home/jifeng 創(chuàng)建目錄 ??mkdir hadoop??

2:解壓

[jifeng@jifeng01 hadoop]$ ls hadoop-1.2.1.tar.gz [jifeng@jifeng01 hadoop]$ tar zxf hadoop-1.2.1.tar.gz [jifeng@jifeng01 hadoop]$ ls hadoop-1.2.1 hadoop-1.2.1.tar.gz [jifeng@jifeng01 hadoop]$
3: 修改hadoop-env.sh配置文件

[jifeng@jifeng01 hadoop]$ cd hadoop-1.2.1 [jifeng@jifeng01 hadoop-1.2.1]$ ls bin hadoop-ant-1.2.1.jar ivy sbin build.xml hadoop-client-1.2.1.jar ivy.xml share c++ hadoop-core-1.2.1.jar lib src CHANGES.txt hadoop-examples-1.2.1.jar libexec webapps conf hadoop-minicluster-1.2.1.jar LICENSE.txt contrib hadoop-test-1.2.1.jar NOTICE.txt docs hadoop-tools-1.2.1.jar README.txt [jifeng@jifeng01 hadoop-1.2.1]$ cd conf [jifeng@jifeng01 conf]$ ls capacity-scheduler.xml hadoop-policy.xml slaves configuration.xsl hdfs-site.xml ssl-client.xml.example core-site.xml log4j.properties ssl-server.xml.example fair-scheduler.xml mapred-queue-acls.xml taskcontroller.cfg hadoop-env.sh mapred-site.xml task-log4j.properties hadoop-metrics2.properties masters [jifeng@jifeng01 conf]$ vi hadoop-env.sh# Set Hadoop-specific environment variables here.# The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes.# The java implementation to use. Required. export JAVA_HOME=/home/jifeng/jdk1.7.0_45# Extra Java CLASSPATH elements. Optional. # export HADOOP_CLASSPATH=# The maximum amount of heap to use, in MB. Default is 1000. # export HADOOP_HEAPSIZE=2000# Extra Java runtime options. Empty by default. # export HADOOP_OPTS=-server# Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS" @ "hadoop-env.sh" 57L, 2436C 已寫入 [jifeng@jifeng01 conf]$ cat hadoop-env.sh

把# ?export JAVA_HOME 修改為“export JAVA_HOME=/home/jifeng/jdk1.7.0_45”

4:修改core-site.xml文件

在hadoop目錄下創(chuàng)建目錄

?[jifeng@jifeng01 hadoop]$ mkdir tmp

[jifeng@jifeng01 conf]$ vi core-site.xml修改后如下：

[jifeng@jifeng01 conf]$ cat core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>fs.default.name</name> <value>hdfs://jifeng01:9000</value> </property><property> <name>hadoop.tmp.dir</name> <value>/home/jifeng/hadoop/tmp</value> </property> </configuration>
5：修改hdfs-site.xml

修改后如下：

[jifeng@jifeng01 conf]$ cat hdfs-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>dfs.replication</name> <value>1</value> <description></description> </property> </configuration>
6:修改mapred-site.xml文件

修改后如下：

[jifeng@jifeng01 conf]$ cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>mapred.job.tracker</name> <value>jifeng01:9001</value> <description>NameNode</description> </property> </configuration> [jifeng@jifeng01 conf]$
7：修改masters和slaves文件

修改后路下

[jifeng@jifeng01 conf]$ cat masters jifeng01 [jifeng@jifeng01 conf]$ cat slaves jifeng02 jifeng03 [jifeng@jifeng01 conf]$
8：先其它2個(gè)節(jié)點(diǎn)復(fù)制hadoop-1.2.1

[jifeng@jifeng01 hadoop]$ scp -r ./hadoop-1.2.1 jifeng02:/home/jifeng/hadoop

[jifeng@jifeng01 hadoop]$ scp -r ./hadoop-1.2.1 jifeng03:/home/jifeng/hadoop

9：格式化分布式文件系統(tǒng)

[jifeng@jifeng01 hadoop-1.2.1]$ bin/hadoop namenode -format 14/07/24 10:29:43 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = jifeng01/10.3.7.214 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.7.0_45 ************************************************************/ 14/07/24 10:29:43 INFO util.GSet: Computing capacity for map BlocksMap 14/07/24 10:29:43 INFO util.GSet: VM type = 64-bit 14/07/24 10:29:43 INFO util.GSet: 2.0% max memory = 932184064 14/07/24 10:29:43 INFO util.GSet: capacity = 2^21 = 2097152 entries 14/07/24 10:29:43 INFO util.GSet: recommended=2097152, actual=2097152 14/07/24 10:29:43 INFO namenode.FSNamesystem: fsOwner=jifeng 14/07/24 10:29:43 INFO namenode.FSNamesystem: supergroup=supergroup 14/07/24 10:29:43 INFO namenode.FSNamesystem: isPermissionEnabled=true 14/07/24 10:29:43 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 14/07/24 10:29:43 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 14/07/24 10:29:43 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 14/07/24 10:29:43 INFO namenode.NameNode: Caching file names occuring more than 10 times 14/07/24 10:29:43 INFO common.Storage: Image file /home/jifeng/hadoop/tmp/dfs/name/current/fsimage of size 112 bytes saved in 0 seconds. 14/07/24 10:29:44 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/home/jifeng/hadoop/tmp/dfs/name/current/edits 14/07/24 10:29:44 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/home/jifeng/hadoop/tmp/dfs/name/current/edits 14/07/24 10:29:44 INFO common.Storage: Storage directory /home/jifeng/hadoop/tmp/dfs/name has been successfully formatted. 14/07/24 10:29:44 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at jifeng01/10.3.7.214 ************************************************************/ [jifeng@jifeng01 hadoop-1.2.1]$
10：啟動(dòng)hadoop

[jifeng@jifeng01 hadoop-1.2.1]$ bin/start-all.sh starting namenode, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-namenode-jifeng01.out jifeng03: starting datanode, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-datanode-jifeng03.out jifeng02: starting datanode, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-datanode-jifeng02.out The authenticity of host 'jifeng01 (10.3.7.214)' can't be established. RSA key fingerprint is a8:9d:34:63:fa:c2:47:4f:81:10:94:fa:4b:ba:08:55. Are you sure you want to continue connecting (yes/no)? yes jifeng01: Warning: Permanently added 'jifeng01,10.3.7.214' (RSA) to the list of known hosts. jifeng@jifeng01's password: jifeng@jifeng01's password: jifeng01: Permission denied, please try again.jifeng01: starting secondarynamenode, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-secondarynamenode-jifeng01.out starting jobtracker, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-jobtracker-jifeng01.out jifeng03: starting tasktracker, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-tasktracker-jifeng03.out jifeng02: starting tasktracker, logging to /home/jifeng/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-jifeng-tasktracker-jifeng02.out [jifeng@jifeng01 hadoop-1.2.1]$ 需要輸入密碼

11：檢測(cè)守護(hù)進(jìn)程

[jifeng@jifeng01 hadoop-1.2.1]$ jps 4539 JobTracker 4454 SecondaryNameNode 4269 NameNode 4667 Jps [jifeng@jifeng01 hadoop-1.2.1]$
[jifeng@jifeng02 hadoop]$ jps 2734 TaskTracker 2815 Jps 2647 DataNode [jifeng@jifeng02 hadoop]$
[jifeng@jifeng03 hadoop]$ jps 4070 Jps 3878 DataNode 3993 TaskTracker [jifeng@jifeng03 hadoop]$

總結(jié)

以上是生活随笔為你收集整理的Hadoop1.2.1集群安装三的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。