hadoop集群搭建步驟

實驗介紹

下面將要在三臺linux虛擬機上搭建hadoop集群。

知識點

linux基本命令

集群安裝

完成實驗需要以下相關知識

解壓命令

tar -zxvf XX.tar.gz -C dist

vi編輯器的使用

vi + file 打開一個文件，要想了解更多請了解vi編輯器的使用

遠程拷貝

scp -r srcfile user@hostName:distpath

實驗前準備

準備三臺linux虛擬機

配置ip和host 下面表格是本次實驗的配置情況

iphost軟件名192.168.1.111linux1java8、hadoop

192.168.1.112linux2java8，hadoop

192.168.1.113linux3java8，hadoop

配置免密登錄，免密登錄方案 linux1免密登錄linux2和linux3

安裝jdk8

準備hadoop2.7.7版本的安裝包

下面開始進行實驗。

hadoop集群搭建實驗

上傳hadoop安裝文件到 /root/apps/srcclauster

進入主節點創建一個目錄apps就作為安裝目錄

[root@linux1 ~]# mkdir /root/apps

復制代碼

解壓hadoop

[root@linux1 ~]#tar –zxvf /root/srcclauster/hadoop-2.7.7.tar.gz -C /root/apps

復制代碼

配置hadoop

進入hadoop配置目錄打開hadoop-env.sh文件 配置一下JAVA_HOME

[root@linux1 ~]#cd /root/srcclauster/hadoop-2.7.7/etc/hadoop

[root@linux1 hadoop]#

[root@linux1 hadoop]# vi hadoop-env.sh

復制代碼

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements. See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership. The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License. You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are

# optional. When running a distributed configuration it is best to

# set JAVA_HOME in this file, so that it is correctly defined on

# remote nodes.

# The java implementation to use.

export JAVA_HOME=/root/appstest1/jdk1.8.0_101

# The jsvc implementation to use. Jsvc is required to run secure datanodes

# that bind to privileged ports to provide authentication of data transfer

# protocol. Jsvc is not required if SASL is configured for authentication of

# data transfer protocol using non-privileged ports.

#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.

for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do

if [ "$HADOOP_CLASSPATH" ]; then

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f

else

export HADOOP_CLASSPATH=$f

fi

done

# The maximum amount of heap to use, in MB. Default is 1000.

#export HADOOP_HEAPSIZE=

#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

# Extra Java runtime options. Empty by default.

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

# Command specific options appended to HADOOP_OPTS when specified

export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"

export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)

export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"

#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"

# On secure datanodes, user to run the datanode as after dropping privileges.

# This **MUST** be uncommented to enable secure HDFS if using privileged ports

# to provide authentication of data transfer protocol. This **MUST NOT** be

# defined if SASL is configured for authentication of data transfer protocol

# using non-privileged ports.

export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}

# Where log files are stored. $HADOOP_HOME/logs by default.

#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER

# Where log files are stored in the secure data environment.

export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###

# HDFS Mover specific parameters

###

# Specify the JVM options to be used when starting the HDFS Mover.

# These options will be appended to the options specified as HADOOP_OPTS

# and therefore may override any similar flags set in HADOOP_OPTS

#

# export HADOOP_MOVER_OPTS=""

###

# Advanced Users Only!

###

# The directory where pid files are stored. /tmp by default.

# NOTE: this should be set to a directory that can only be written to by

# the user that will run the hadoop daemons. Otherwise there is the

# potential for a symlink attack.

export HADOOP_PID_DIR=${HADOOP_PID_DIR}

export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default.

export HADOOP_IDENT_STRING=$USER

復制代碼

打開core-site.xml文件配置一下主節點和工作目錄

[root@linux1 hadoop]# vi core-site.xml

復制代碼

fs.defaultFS

hdfs://linux1:9000

hadoop.tmp.dir

/root/appstest1/appdata

復制代碼

打開mapred-site.xml配置MR運行方式

[root@linux1 hadoop]# vi mapred-site.xm

復制代碼

mapreduce.framework.name

yarn

復制代碼

打開yarn-site.xml文件配置yarn的主節點

[root@linux1 hadoop]# vi yarn-site.xml

復制代碼

yarn.resourcemanager.hostname

linux1

yarn.nodemanager.aux-services

mapreduce_shuffle

復制代碼

配置slaves

[root@linux1 hadoop]# vi slaves

復制代碼

linux2

linux3

復制代碼

格式化hdfs

[root@linux1 ~]#/root/hadoop-2.7.7/bin/hadoop namenode -format

復制代碼

啟動hadoop集群

進入linux1

[root@linux1 apps]# /root/apps/hadoop-2.7.7/sbin/start-dfs.sh

20/04/27 16:14:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [linux1]

linux1: starting namenode, logging to /root/apps/hadoop-2.7.7/logs/hadoop-root-namenode-linux1.out

linux3: datanode running as process 1618. Stop it first.

linux2: datanode running as process 1617. Stop it first.

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /root/apps/hadoop-2.7.7/logs/hadoop-root-secondarynamenode-linux1.out

20/04/27 16:15:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[root@linux1 apps]#

復制代碼

測試是否啟動成功

總結

配置核心4個文件 ，hadoop-env.sh配置JAVA_HOME，core-site.xml配置主節點，mapred-site.xm配置MR運行方式， yarn-site.xml配置yarn的主節點。

總結

以上是生活随笔為你收集整理的linux hadoop集群搭建,hadoop集群搭建的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。