CDH Hadoop基于Cloudera Manager半在線方式安裝詳解

1 CM簡(jiǎn)介

Cloudera Manager是一個(gè)端到端的管理CDH集群的工具。它通過管理CDH集群（組件安裝、服務(wù)管理、參數(shù)監(jiān)控、問題診斷、性能監(jiān)控）各個(gè)環(huán)節(jié)從而增強(qiáng)集群管理的質(zhì)量, 增強(qiáng)規(guī)范性,降低運(yùn)營(yíng)成本。

注:CM自6.3.3之后不再支持社區(qū)免費(fèi)版(Express),后續(xù)版本需付費(fèi)才能下載和使用。

2 準(zhǔn)備

2.1 環(huán)境說明

1 該安裝方式依賴如下linux組件:httpd, mysql

2 安裝時(shí)需要用到其它系統(tǒng)組件,請(qǐng)配置好本地源

3 安裝mysql是請(qǐng)確保清理好系統(tǒng)自帶的mysql.

4 CM 下載地址見下,需要下載如下的所有文件.詳細(xì)目錄分布見2.2.6說明.

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/RPMS/x86_64/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera

http://archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo

5 CDH parcel下載地址http://archive-primary.cloudera.com/cdh5/parcels/5.5.1/

6 CM installer下載地址

http://archive-primary.cloudera.com/cm5/installer/5.5.1/cloudera-manager-installer.bin

注:

1 CM 核心安裝文件截圖:

2 CDH parcel相關(guān)文件截圖:

?

3 CM installer截圖:

2.1.1 虛擬機(jī)里L(fēng)inux系統(tǒng)版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

?

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官網(wǎng)強(qiáng)調(diào)一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit ServerVM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

+-----------+

| version() |

+-----------+

| 5.6.14 ???|

+-----------+

注:

1 Mysql 驅(qū)動(dòng)為mysql-connector-java-5.1.31-bin.jar

2 該jar需要放置在/usr/share/cmf/lib/內(nèi)被CM安裝時(shí)用到

2.2 安裝準(zhǔn)備

2.2.1 禁用防火墻

service iptables stop

chkconfig iptables off

2.2.2 禁用Selinux

vi /etc/selinux/config

#修改內(nèi)容見下:

SELINUX=disabled

SELINUXTYPE=targeted

2.2.3 修改主機(jī)名

vi/etc/hosts

#加上archive.cloudera.com域名

192.168.56.101 archive.cloudera.com shenl

vi/etc/sysconfig/network

#修改主機(jī)名如下:

HOSTNAME=archive.cloudera.com

#重啟電腦使得更改生效.

2.2.4 配置本地源

1) 虛擬機(jī)加載Linux ISO鏡像

2) 找到rom對(duì)應(yīng)設(shè)備名

[root@localhost ~]# lsblk

NAME??????????????????????? MAJ:MIN RM? SIZE RO TYPE MOUNTPOINT

sr0????????????????????????? 11:0??? 1 1024M?0 rom?

sr1????????????????????????? 11:1??? 1?3.6G? 0 rom?

sr2????????????????????????? 11:2??? 1 1024M?0 rom?

sda?????????????????????????? 8:0??? 0???8G? 0 disk

?..sda1??????????????????????? 8:1??? 0?500M? 0 part /boot

?..sda2??????????????????????? 8:2??? 0?7.5G? 0 part

??..VolGroup-lv_root (dm-0) 253:0???0? 6.7G? 0 lvm?/

??..VolGroup-lv_swap (dm-1) 253:1???0? 816M? 0 lvm?[SWAP]

3) 創(chuàng)建目錄

[root@localhost ~]#mkdir -p /mnt/cdrom/

4) 掛載鏡像

mount -t iso9660 /dev/sr1 /mnt/cdrom

5) 配置linux本地更新源(真實(shí)環(huán)境這里的應(yīng)該baseurl應(yīng)配置成http方式,如http://ios)

vi /etc/yum.repos.d/redhat.repo

#修改內(nèi)容如下:

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum makecache

yum update list

2.2.5 安裝配置mysql

1)? 刪除系統(tǒng)自帶的mysql

rpm -qa | grep -i mysql

MySQL-client-5.1.49-1.glibc23

?

rpm -e --nodeps MySQL-client-5.1.49-1.glibc23

chkconfig --list | grep -i mysql?

mysql??????????0:off?? 1:off?? 2:on???3:on??? 4:on??? 5:on???6:off?

chkconfig --del mysql

whereis mysql?

mysql: /usr/lib/mysql /usr/share/mysql ?

rm -rf /usr/lib/mysql/?

rm -rf /usr/share/mysql

2) 安裝Mysql服務(wù)、客戶端

rpm -ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm -ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

3) 啟動(dòng)mysql服務(wù)

service mysql start

4) 修改mysql初始密碼,先找到安裝時(shí)的初始密碼,在修改成自己的密碼

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql -uroot -p RUmKBqcY

set password=password(‘root’)

注:??如果想修改mysql默認(rèn)的datadir,假設(shè)到/data/mysqldata內(nèi),可參考如下方式修改:

#遷移mysql數(shù)據(jù)目錄,做軟連接,修改目錄的所有者

mv /var/lib/mysql//data/mysqldata/mysql????????????????

ln -s /data/mysqldata/mysql/var/lib/

chown -R mysql:mysql/data/mysqldata/mysql

2.2.6 安裝httpd,并按照如下的目錄結(jié)構(gòu)放置CM和CDH parcel的安裝文件.

yum install tree

yum install httpd

service httpd start

cd /var/www/html

注:如果想修改httpd的網(wǎng)站默認(rèn)主目錄,可通過如下方式:

mkdir -p /data/www/

chown -R apache:apache/data/www/

vi /etc/httpd/conf/httpd.conf

#找到DocumentRoot 修改"/var/www/html"成"/data/www"

service httpd restart

報(bào)如下錯(cuò):

Forbidden

You don't have permission to access /cm/ on this server.

vi /etc/httpd/conf/httpd.conf

#找到 <Directory"/var/www/html"> 修改成 /data/www

service httpd restart

#CM的目錄安排需要按照如下方式:

[root@archive html]# tree./cm5/

./cm5/

└── redhat

???└── 6

???????└── x86_64

??????????? └── cm

??????????????? ├── 5.5.1

??????????????? │??├── repodata

??????????????? │??│??├── filelists.xml.gz

??????????????? │??│??├── filelists.xml.gz.asc

??????????????? │??│??├── other.xml.gz

??????????????? │??│??├── other.xml.gz.asc

??????????????? │??│??├── primary.xml.gz

??????????????? │??│??├── primary.xml.gz.asc

??????????????? │??│??├── repomd.xml

??????????????? │??│??└── repomd.xml.asc

??????????????? │??└── RPMS

??????????????? │??????└── x86_64

??????????????? │??????????├──cloudera-manager-agent-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

???????????????│??????????├── cloudera-manager-daemons-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──cloudera-manager-server-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──cloudera-manager-server-db-2-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├──enterprise-debuginfo-5.5.1-1.cm551.p0.8.el6.x86_64.rpm

??????????????? │??????????├── jdk-6u31-linux-amd64.rpm

??????????????? │??????????└──oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

??????????????? ├── cloudera-manager.repo

??????????????? └── RPM-GPG-KEY-cloudera

8 directories, 17 files

#CDH parcel的目錄安排需要按照如下方式:

[root@archive html]#tree ./cdh5/

./cdh5/

└── parcels

???└── 5.5.1

???????├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel

???????├── CDH-5.5.1-1.cdh5.5.1.p0.11-el6.parcel.sha1

???????└── manifest.json

?

2 directories, 3 files

#將CM安裝腳本放置/var/www/html(可放置在其它目錄,這里為了統(tǒng)一).:

chmod? +x cloudera-manager-installer.bin

注:下載repodata文件夾時(shí)可以在Window里下載好,執(zhí)行wget命令

E:\ProgramFiles\wget-1.16.3-win64>wget -r -np -ndhttp://archive-primary.cloude

ra.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/

3安裝

3.1安裝中

1) 執(zhí)行CM安裝腳本

./cloudera-manager-installer.bin

#分析并解決CM安裝過程中的問題,詳細(xì)見 3.2安裝問題:

cat/var/log/cloudera-manager-installer/2.install-oracle-j2sdk1.7.log

2) 輸入CM所在的IP地址和7180端口,進(jìn)行界面化安裝CDH.

?#選擇存儲(chǔ)庫(kù)時(shí)需要點(diǎn)擊parcel旁邊的更多選項(xiàng),刪除默認(rèn)定義的路徑,寫上

http://192.168.56.101/cdh5/parcels/5.5.1

或者

http://archive.cloudera.com/cdh5/parcels/5.5.1

#這時(shí)如果選擇此Cloudera Manager Server的匹配發(fā)行版,會(huì)報(bào)錯(cuò),想見3.2安裝問題之d).這里建議選擇自定義存儲(chǔ)庫(kù).見下圖:

2)? 按照提示一步步安裝,直至集群安裝成功…

注: CM在安裝時(shí)會(huì)自動(dòng)生成cloudera-manager.repo,請(qǐng)確認(rèn)里面的url跟自定義存儲(chǔ)庫(kù)里的一致,建議走h(yuǎn)ttp協(xié)議.cloudera-manager.repo里默認(rèn)下載地址見下:

https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1

3.2 安裝問題

1) 正確配置CM和CDH的目錄樹結(jié)構(gòu)

2) 正確配置parcel和自定義存儲(chǔ)庫(kù)

a) Loaded plugins: product-id,security, subscription-manager

This system is not registered to Red Hat SubscriptionManagement. You can use subscription-manager to register.

http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/repomd.xml:[Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 NotFound"

解決方案:重命名文件名,詳細(xì)見下:

cd /var/www/html

mv ./cm5/redhat/6/x86_64/cm/5.5.1./cm5/redhat/6/x86_64/cm/5

b) .

解決方案:重命名文件名,詳細(xì)見如下:

mv ./cm5/redhat/6/x86_64/cm/5 ./cm5/redhat/6/x86_64/cm/5.5.1

c) CMinstaller正常安裝相關(guān)日志

?[root@archive cloudera-manager-installer]# pwd

/var/log/cloudera-manager-installer

[root@archivecloudera-manager-installer]# ll

total 20

-rw-r--r-- 1 root root??? 0 Jan?4 15:51 0.check-selinux.log

-rw-r--r-- 1 root root??? 0 Jan?4 15:51 1.install-repo-pkg.log

-rw-r--r-- 1 root root 1647 Jan? 4 15:51 2.install-oracle-j2sdk1.7.log

-rw-r--r-- 1 root root 2080 Jan? 4 15:53 3.install-cloudera-manager-server.log

-rw-r--r-- 1 root root??? 0 Jan?4 15:46 3.remove-cloudera-manager-repository.log

-rw-r--r-- 1 root root 3250 Jan? 4 15:53 4.install-cloudera-manager-server-db-2.log

-rw-r--r-- 1 root root 1978 Jan? 4 15:54 5.start-embedded-db.log

-rw-r--r-- 1 root root?? 59 Jan?4 15:54 6.start-scm-server.log

d) 按照錯(cuò)誤提示修改成b)后仍然報(bào)如下錯(cuò)誤

BEGIN yum info cloudera-manager-agent?
Loaded plugins: product-id, security, subscription-manager?
This system is not registered to Red Hat Subscription Management. You can usesubscription-manager to register.?
https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.5.1/repodata/repomd.xml:[Errno 14] PYCURL ERROR 7 - "couldn't connect to host"?
Trying other mirror.?
Error: Cannot retrieve repository metadata (repomd.xml) for repository:cloudera-manager. Please verify its path and try again

解決方案:這時(shí)需要在 群集安裝→選擇存儲(chǔ)庫(kù)里選擇自定義存儲(chǔ)庫(kù),配置見下圖:

?

e) 配置CM事件監(jiān)控、報(bào)表管理數(shù)據(jù)庫(kù)時(shí)的錯(cuò)誤

解決方案: 拷貝mysql驅(qū)動(dòng)到cm的相關(guān)lib目錄下.

cp mysql-connector-java-5.1.31-bin.jar/usr/share/cmf/lib/

注:mysql驅(qū)動(dòng)事先下載好.

f) 添加Hive服務(wù)指定mysql作為數(shù)據(jù)源時(shí)報(bào)找不到驅(qū)動(dòng)的錯(cuò)誤

解決方法:拷貝mysql驅(qū)動(dòng)

cp mysql-connector-java-5.1.31-bin.jar/usr/share/java/

mv /usr/share/java/mysql-connector-java.jar

g) CM集群驗(yàn)證錯(cuò)誤

解決方法:按照提示修改告警

vi/etc/sysctl.conf

vm.swappiness= 0

/sbin/sysctl-p /etc/sysctl.conf

#確認(rèn)是否修改為0

cat/proc/sys/vm/swappiness

#永久修復(fù)該告警

vi/etc/rc.local

#追加如下內(nèi)容到上述文件內(nèi)
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

#臨時(shí)修復(fù)該告警

echonever > /sys/kernel/mm/redhat_transparent_hugepage/defrag

4 配置

暫無

5 使用

登錄CM安裝IP和7180對(duì)CDH進(jìn)行管理和維護(hù).

6 卸載

暫無

7 總結(jié)

通過日志分析安裝的問題.

總結(jié)

以上是生活随笔為你收集整理的CDH Hadoop 基于CM方式半在线安装步骤详解的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。