你還在為內存不夠用而發愁?
感覺不感覺臺式機用著不順手?

參看本教程只需四步讓你開發的web項目不再依賴自己的HDFS集群
本文詳細介紹自己電腦上開發的項目如何連接其他機器的HDFS集群
最終成果:
團隊內共用一個HDFS集群開發項目
條件:
同一局域網內

第一步:

參看我的局域網下用其他電腦連接VM虛擬機方法
這個是必須的,如果你會了那可以跳過

第二步:

學會怎么配置虛擬網路之后我們看一下這個配置文件
hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.nameservices</name><value>mycluster</value></property><property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property><property> <name>dfs.namenode.rpc-address.mycluster.nn1</name><value>node01:8020</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>node02:8020</value></property><property><name>dfs.namenode.http-address.mycluster.nn1</name><value>node01:50070</value></property><property><name>dfs.namenode.http-address.mycluster.nn2</name><value>node02:50070</value></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value></property><property><name>dfs.journalnode.edits.dir</name><value>/var/abc/hadoop/ha/jn</value></property><property><name>dfs.client.failover.proxy.provider.mycluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property></configuration>

第三步 修改虛擬機和本機的Ip端口映射

可以看到上面有許多端口號,是虛擬機的端口.我們去VMware修改這些端口的映射
修改方法參看:局域網下用其他電腦連接VM虛擬機方法

如此一來我們就把虛擬機node01的50070端口映射為本機的50071端口下了,同樣的我們將上面代碼所有的端口全部用本機端口映射一遍,




這里就不全部貼出來了,修改完成之后

第四步 修改配置文件

為本機的Ip加對應的端口號

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.nameservices</name><value>mycluster</value></property><property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property><property> <name>dfs.namenode.rpc-address.mycluster.nn1</name><value>172.17.36.93:18020</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>172.17.36.93:28020</value></property><property><name>dfs.namenode.http-address.mycluster.nn1</name><value>172.17.36.93:50073</value></property><property><name>dfs.namenode.http-address.mycluster.nn2</name><value>172.17.36.93:50072</value></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://172.17.36.93:18485;172.17.36.93:28485;172.17.36.93:38485/mycluster</value></property><property><name>dfs.journalnode.edits.dir</name><value>/var/abc/hadoop/ha/jn</value></property><property><name>dfs.client.failover.proxy.provider.mycluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property></configuration>

到這里還沒完,還有一個配置文件需要修改
core-site.xml
同樣的操作,將原來2181端口映射為本機的端口,修改配置文件為

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property> <name>fs.defaultFS</name><value>hdfs://mycluster</value></property><property><name>ha.zookeeper.quorum</name><value>172.17.36.93:21812,172.17.36.93:21813,172.17.36.93:21814</value></property></configuration>

完成上述操作后在本機啟動你的web項目已經和原來一樣可以用了
但是用其他電腦運行web項目仍然報404什么原因呢?
答案是本機的防火墻需要放出剛才設置的那些端口,或者直接將防火墻關閉

如此一來,在同一局域網下的所有電腦,只要帶著你修改后的配置文件,就可以訪問你的HDFS集群了.

感謝您的瀏覽

總結

以上是生活随笔為你收集整理的局域网下连接其他电脑的HDFS集群的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。