?

今天在hadoop集群環(huán)境下需要將兩臺(tái)datanode刪除，為了不影響在運(yùn)行業(yè)務(wù)，需對(duì)節(jié)點(diǎn)進(jìn)行動(dòng)態(tài)刪除，記錄操作過程如下：

?

1， 從集群中移走節(jié)點(diǎn)，需要對(duì)移走節(jié)點(diǎn)的數(shù)據(jù)進(jìn)行備份：

在主節(jié)點(diǎn)的core-site.xml配置文件中添加如下內(nèi)容：

<property>

? ? ? ? ? <name>dfs.hosts.exclude</name>

? ? ? ? ? <value>/home/hadoop/hadoop/conf/excludes</value>

</property>

?

說明

dfs.hosts.exclude：指要?jiǎng)h除的節(jié)點(diǎn)

/home/hadoop/hadoop/conf/excludes：指定要被刪除文件所在路徑及名稱，該處定義為excludes

?

2， 在1中設(shè)置目錄中touch excludes，內(nèi)容為每行需要移走的節(jié)點(diǎn)

cloud4

cloud5

?

3，進(jìn)入 運(yùn)行命令：hadoop dfsadmin -refreshNodes（我這用的yum安裝的，不同安裝方式hadoop目錄會(huì)在不同路徑），該命令可以動(dòng)態(tài)刷新dfs.hosts和dfs.hosts.exclude配置，無需重啟NameNode。

?執(zhí)行完成被刪除節(jié)點(diǎn)datanode消失了，但是tasktracker還會(huì)存在，需要自己手動(dòng)停掉

?

4，然后通過 bin/hadoop dfsadmin -report查看，結(jié)果如下：?

?

Configured Capacity: 17721082527744 (16.12 TB)

Present Capacity: 16806607028262 (15.29 TB)

DFS Remaining: 14996775104512 (13.64 TB)

DFS Used: 1809831923750 (1.65 TB)

DFS Used%: 10.77%

Under replicated blocks: 6788

Blocks with corrupt replicas: 0

Missing blocks: 0

?

-------------------------------------------------

Datanodes available: 6 (6 total, 0 dead)

?

Name: 192.168.1.5:50010

Decommission Status : Normal

Configured Capacity: 2953511657472 (2.69 TB)

DFS Used: 265079108972 (246.87 GB)

Non DFS Used: 150286670484 (139.97 GB)

DFS Remaining: 2538145878016(2.31 TB)

DFS Used%: 8.98%

DFS Remaining%: 85.94%

Last contact: Thu Sep 08 10:12:45 CST 2011

?

Name: 192.168.1.8:50010

Decommission Status : Decommission in progress

Configured Capacity: 2953511657472 (2.69 TB)

DFS Used: 228590288896 (212.89 GB)

Non DFS Used: 150240718848 (139.92 GB)

DFS Remaining: 2574680649728(2.34 TB)

DFS Used%: 7.74%

DFS Remaining%: 87.17%

Last contact: Thu Sep 08 10:12:45 CST 2011

?

?Name: 192.168.1.7:50010

Decommission Status : Normal

Configured Capacity: 2953511657472 (2.69 TB)

DFS Used: 266826599821 (248.5 GB)

Non DFS Used: 150259458675 (139.94 GB)

DFS Remaining: 2536425598976(2.31 TB)

DFS Used%: 9.03%

DFS Remaining%: 85.88%

Last contact: Thu Sep 08 10:12:46 CST 2011

?

?Name: 192.168.1.9:50010

Decommission Status : Decommission in progress

Configured Capacity: 2953511657472 (2.69 TB)

DFS Used: 226060701696 (210.54 GB)

Non DFS Used: 150240718848 (139.92 GB)

DFS Remaining: 2577210236928(2.34 TB)

DFS Used%: 7.65%

DFS Remaining%: 87.26%

Last contact: Thu Sep 08 10:12:45 CST 2011

?

?Name: 192.168.1.4:50010

Decommission Status : Normal

Configured Capacity: 2953524240384 (2.69 TB)

DFS Used: 553202110857 (515.21 GB)

Non DFS Used: 163197603447 (151.99 GB)

DFS Remaining: 2237124526080(2.03 TB)

DFS Used%: 18.73%

DFS Remaining%: 75.74%

Last contact: Thu Sep 08 10:12:46 CST 2011

?

?Name: 192.168.1.6:50010

Decommission Status : Normal

Configured Capacity: 2953511657472 (2.69 TB)

DFS Used: 270073113508 (251.53 GB)

Non DFS Used: 150250329180 (139.93 GB)

DFS Remaining: 2533188214784(2.3 TB)

DFS Used%: 9.14%

DFS Remaining%: 85.77%

Last contact: Thu Sep 08 10:12:44 CST 2011

?

?

?

5，通過4中命令可以查看到被刪除節(jié)點(diǎn)狀態(tài)，如192.168.1.9

?

Decommission Status : Decommissioned

說明從91往其他節(jié)點(diǎn)同步數(shù)據(jù)已經(jīng)完成，如果狀態(tài)為Decommission Status : Decommissione in process則還在執(zhí)行。

?

?

?

至此刪除節(jié)點(diǎn)操作完成

?

?

?

問題總結(jié)

?

在拔掉節(jié)點(diǎn)時(shí)注意要把往hadoop放數(shù)據(jù)程序先停掉，否則程序還會(huì)往要?jiǎng)h除節(jié)點(diǎn)同步數(shù)據(jù)，刪除節(jié)點(diǎn)程序會(huì)一直執(zhí)行。

轉(zhuǎn)載于:https://www.cnblogs.com/licheng/archive/2011/11/08/2241854.html

總結(jié)

以上是生活随笔為你收集整理的hadoop删除节点的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。