【聯邦學習】FATE 集群部署step1
【聯邦學習】FATE 集群部署step2

• 部署FATE

參考中文集群部署文檔

all 指的是2個虛擬機都要操作
f1 只的是1號虛擬機
f2 指的是2號虛擬機

f1 root

hostnamectl set-hostname VM_0_1_centos

f2 root

hostnamectl set-hostname VM_0_2_centos

all root

vim /etc/hosts加入主機映射 追加 192.168.0.1 VM_0_1_centos 192.168.0.2 VM_0_2_centos3.2 關閉selinux sed -i '/^SELINUX/s/=.*/=disabled/' /etc/selinux/config setenforce 03.3 修改Linux最大打開文件數 vim /etc/security/limits.conf 追加 * soft nofile 65536 * hard nofile 655363.4 關閉防火墻 systemctl disable firewalld.service systemctl stop firewalld.service systemctl status firewalld.service3.5 軟件環境初始化 mkdir -p /data/app cd /data/app wget https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/fate-base.tar.gz sudo tar -xvzf fate-base.tar.gz cd fate-base3.5.1 初始化服務器 1）初始化服務器 sudo sh env.sh chown –R app:apps /data/app2）配置sudo vim /etc/sudoers.d/app 新增 app ALL=(ALL) ALL app ALL=(ALL) NOPASSWD: ALL Defaults !env_reset 3）配置ssh無密登錄 su app 可能要設置密碼 默認建議輸入1ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub >> /home/app/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keysb.合并id_rsa_pub文件f1 app scp ~/.ssh/authorized_keys app@192.168.0.2:/home/app/.sshf2 app cat ~/.ssh/id_rsa.pub >> /home/app/.ssh/authorized_keys scp ~/.ssh/authorized_keys app@192.168.0.1:/home/app/.sshall app ssh app@192.168.0.1 ssh app@192.168.0.2 連接測試all app 安裝時間較長 需要耐心等待 不要亂動 sh install_java.sh sh install_redis.sh sh install_py3.sh sh install_mysql.sh 這里安裝好了之后要 lsof-i:3306 *看下mysql 的狀態 保證在運行 重要 這里是坑 不然后面會蹦/data/projects/common/mysql/mysql-8.0.13/bin/mysql -uroot -p -S /data/projects/common/mysql/mysql-8.0.13/mysql.sock輸入 fate_dev 進入mysql set password='fate_dev'; CREATE USER 'root'@'192.168.0.1' IDENTIFIED BY 'fate_dev'; GRANT ALL ON *.* TO 'root'@'192.168.0.1'; CREATE USER 'fate_dev'@'192.168.0.1' IDENTIFIED BY 'fate_dev'; GRANT ALL ON *.* TO 'fate_dev'@'192.168.0.1'; CREATE USER 'root'@'192.168.0.2' IDENTIFIED BY 'fate_dev'; GRANT ALL ON *.* TO 'root'@'192.168.0.2'; CREATE USER 'fate_dev'@'192.168.0.2' IDENTIFIED BY 'fate_dev'; GRANT ALL ON *.* TO 'fate_dev'@'192.168.0.2'; CREATE USER 'fate_dev'@'localhost' IDENTIFIED BY 'fate_dev'; GRANT ALL ON *.* TO 'fate_dev'@'localhost'; flush privileges; * 注意和官方文檔不一樣 需要多添加一些權限 不然后面會報錯f1 app cd /data/projects/ 這里可能較慢 git clone https://github.com/WeBankFinTech/FATE.git* 和官方文檔的不同 此處需安裝maven sudo wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.zip sudo unzip apache-maven-3.6.2-bin.zip sudo vim /etc/profile 追加 export MAVEN_HOME=/data/projects/apache-maven-3.6.2 export PATH=${MAVEN_HOME}/bin:$PATH生效 source /etc/profile改阿里云鏡像 為了加速下載 sudo vim /data/projects/apache-maven-3.6.2/conf/settings.xml 找到mirrors元素， 在它里面添加子元素mirror： <mirror><id>nexus</id><mirrorOf>*</mirrorOf> <url>http://maven.aliyun.com/nexus/content/groups/public/</url></mirror><mirror><id>nexus-public-snapshots</id><mirrorOf>public-snapshots</mirrorOf><url>http://maven.aliyun.com/nexus/content/repositories/snapshots/</url> </mirror>查看是否安裝成功 mvn -v*和官方文檔的不同 此處需安裝java套件 安裝java套件 sudo yum install java-devel進入項目的arch目錄，進行構建打包： cd FATE/arch mvn clean package -DskipTests cd ../fate-serving mvn clean package -DskipTests cd ../fateboard mvn clean package -DskipTestscd /data/projects wget https://webank-ai-1251170195.cos.ap-guangzhou.myqcloud.com/third_party.tar.gz sudo tar -xzvf third_party.tar.gz -C FATE/arch/eggroll/storage-service-cxx/third_partysudo vim /data/projects/FATE/cluster-deploy/scripts/configurations.sh 新增 #!/bin/bash user=app dir=/data/projects/fate mysqldir=/data/projects/common/mysql/mysql-8.0.13 javadir=/data/projects/common/jdk/jdk1.8.0_192 venvdir=/data/projects/fate/venv redisip=(192.168.0.1 192.168.0.2) redispass=fate_dev partylist=(10000 9999) JDBC0=(192.168.0.1 eggroll_meta fate_dev fate_dev) JDBC1=(192.168.0.2 eggroll_meta fate_dev fate_dev) fateflowdb0=(192.168.0.1 fate_flow fate_dev fate_dev) fateflowdb1=(192.168.0.1 fate_flow fate_dev fate_dev) iplist=(192.168.0.1 192.168.0.2) iplist0=(192.168.0.1) iplist1=(192.168.0.2) fateboard0=(192.168.0.1) fateboard1=(192.168.0.2) eggautocompile=true fedlist0=(192.168.0.1) fedlist1=(192.168.0.2) meta0=(192.168.0.1) meta1=(192.168.0.2) proxy0=(192.168.0.1) proxy1=(192.168.0.2) roll0=(192.168.0.1) roll1=(192.168.0.2) egglist0=(192.168.0.1) egglist1=(192.168.0.2) fllist0=(192.168.0.1) fllist1=(192.168.0.2) serving0=(192.168.0.1) serving1=(192.168.0.2) exchangeip=運行打包部署 cd /data/projects/FATE/cluster-deploy/scripts sudo bash auto-packaging.sh sudo bash auto-deploy.sh 注意這里有點慢 而且要輸入很多次 f1 和 f2 app用戶的密碼all app 啟動服務cd /data/projects/fate sh services.sh all startcd /data/projects/fate/serving-server sh service.sh startcd /data/projects/fate/python/fate_flow sh service.sh start檢查服務狀態 cd /data/projects/fate sh services.sh all statuscd /data/projects/fate/serving-server sh service.sh statuscd /data/projects/fate/python/fate_flow sh service.sh status 一定要保證這個服務啟動了 不然查看錯誤日志注意這里如果報錯沒有找到對應模塊 需要添加 執行模塊到 python path最小化測試 f1 app export PYTHONPATH = /data/projects/fate/python source /data/projects/fate/venv/bin/activate cd /data/projects/fate/python/examples/min_test_task /sh run.sh host fastf2 app export PYTHONPATH = /data/projects/fate/python source /data/projects/fate/venv/bin/activate cd /data/projects/fate/python/examples/min_test_task/這里直接把$ {host_table} $ {host_namespace} 換成f1 中答應出來的兩個變量就可以了sh run.sh guest fast $ {host_table} $ {host_namespace}

總結

以上是生活随笔為你收集整理的【联邦学习】FATE 集群部署 step3的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。