联邦学习fate笔记小结
20211006
https://gitee.com/jfdwd/FedRec
fate推薦
faterec
job_config和task_config不是同一個東西
task_config包含job_config里面的一些參數(shù)
20210930
導(dǎo)包錯誤
這里的導(dǎo)包錯誤是因?yàn)榘裮iniconda變成了source_root
取消之后就好了
no module named requests 錯誤是沒有在服務(wù)器上面把項(xiàng)目所在地的上一層目錄設(shè)置為環(huán)境變量
在pycharm中設(shè)置為content_root的意思就是把這個路徑加入系統(tǒng)環(huán)境變量
一個項(xiàng)目中只能有一個content_root 可以是多個source_folders
ModuleNotFoundError
: No module named "arch": 將PYTHONPATH設(shè)置為fate_flow目錄的父目錄。
20210922
lmdb.Error: Attempt to operate on closed/deleted/dropped object.
把分區(qū)數(shù)設(shè)小一點(diǎn) 內(nèi)存不夠 最好把之前上傳的數(shù)據(jù)物理刪掉之后再重新
上傳
https://github.com/FederatedAI/FedRec/blob/master/fate_flow/README_zh.md
運(yùn)行流程fateflow
https://fate.readthedocs.io/en/latest/_build_temp/examples/dsl/v1/README.html?highlight=download
下載預(yù)測結(jié)果
conf文件兩個地方其實(shí)是同一個東西
test 的地方只是給個案例給人參照看的
component homo_lr_o
module： homoLR
20210917
Parameter object is the only way to pass user-define runtime parameters to the developing module, so every module has it’s own parameter object. In order to define a usable parameter object, three steps will be needed.
The purpose to define a setting conf is that fate_flow module extract this file to get the information of how to start program of the module.
upstream components(DataIO for example)
https://fate.readthedocs.io/en/latest/_build_temp/doc/develop_guide.html
開發(fā)指南
flow data upload -c /fate-root/fate/examples/dsl/v1/homo_logistic_regression/upload_data_guest.json上傳數(shù)據(jù)的時候用完整的絕對路徑
具體執(zhí)行代碼的模塊
task_executor.py
-j 202109170938469173010 -n hetero_lr_0 -t 202109170938469173010_hetero_lr_0 -v 0 -r host -p 10000 -c /fate-root/fate/jobs/202109170938469173010/host/10000/hetero_lr_0/202109170938469173010_hetero_lr_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380-j 202109170938469173010 -n hetero_lr_0 -t 202109170938469173010_hetero_lr_0 -v 0 -r guest -p 9999 -c /fate-root/fate/jobs/202109170938469173010/guest/9999/hetero_lr_0/202109170938469173010_hetero_lr_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380模型部分 task_executor的參數(shù)-j 202109170938469173010 -n dataio_0 -t 202109170938469173010_dataio_0 -v 0 -r guest -p 9999 -c /fate-root/fate/jobs/202109170938469173010/guest/9999/dataio_0/202109170938469173010_dataio_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380-j 202109170938469173010 -n dataio_0 -t 202109170938469173010_dataio_0 -v 0 -r host -p 10000 -c /fate-root/fate/jobs/202109170938469173010/host/10000/dataio_0/202109170938469173010_dataio_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380dataio task_executor的參數(shù)只需修改對應(yīng)部分的參數(shù)就行了20210915
需要進(jìn)入細(xì)節(jié)研究的是我們的每個任務(wù)模塊(fate中稱為component)，比如（intersect,featurebinning,hetero_lr等），所以我們?nèi)艄雂ebug，必須找到fate-flow-server執(zhí)行每個模塊的啟動代碼（或者說執(zhí)行入口）
與具體每個component的執(zhí)行入口最相關(guān)的兩個文件 task_scheduler.py 和task_executor.py
看文件名我們就很清楚知道，第一個是task（這里起名為task，是需要和job區(qū)分，job主要是指我們整體的作業(yè)，而task是根據(jù)job配置文件拆分后得到的每個子任務(wù)）調(diào)度相關(guān)代碼，第二個是task執(zhí)行相關(guān)的代碼
https://zhuanlan.zhihu.com/p/105798559
https://zhuanlan.zhihu.com/p/342079952?ivk_sa=1024320u
聯(lián)邦學(xué)習(xí)開源框架FATE研究–使用PyCharm開發(fā)和調(diào)試
C:\Users\hz\AppData\Local\JetBrains\PyCharm2020.1\remote_sources\4510329\-329753495\pipeline
pipeline 運(yùn)行時所需要的文件所在位置
{"data":{"board_url":"http://127.0.0.1:8080/index.html#/dashboard?job_id=202109151539074672960&role=guest&party_id=10000","job_dsl_path":"/fate-root/fate/jobs/202109151539074672960/job_dsl.json","job_id":"202109151539074672960","job_runtime_conf_on_party_path":"/fate-root/fate/jobs/202109151539074672960/guest/job_runtime_on_party_conf.json","job_runtime_conf_path":"/fate-root/fate/jobs/202109151539074672960/job_runtime_conf.json","logs_directory":"/fate-root/fate/logs/202109151539074672960","model_info":{"model_id":"arbiter-9999#guest-10000#host-9999#model","model_version":"202109151539074672960"},"pipeline_dsl_path":"/fate-root/fate/jobs/202109151539074672960/pipeline_dsl.json","train_runtime_conf_path":"/fate-root/fate/jobs/202109151539074672960/train_runtime_conf.json"},"jobId":"202109151539074672960","retcode":0,"retmsg":"success"}運(yùn)行過程中各種文件信息的位置D:\project_pycharm\fate-test\venv\lib\python3.6\site-packages\flow_sdk\client\base.py
這個文件里面的 response 里面DSL:就是根據(jù)pipeline流程自動生成的一個流程配置文件
components就等于整個DSL 文件
通過model_info 來檢索模型
model_id和model_version
submit_conf
initiator
role
job_parameters
componets_parameters
1 File "./fate/python/fate_flow/operation/task_executor.py", line 154, in run_task
12 run_object.run(component_parameters_on_party, task_run_args)
13 File "./fate/python/federatedml/model_base.py", line 98, in run
14 this_data_output = func(*real_param)
15 File "./fate/python/federatedml/util/data_transform.py", line 885, in fit
16 data_inst = self.transformer.read_data(data_inst, "fit")
17 File "./fate/python/federatedml/util/data_transform.py", line 120, in read_data
18 abnormal_detection.empty_table_detection(input_data)
19 File "./fate/python/federatedml/util/abnormal_detection.py", line 29, in empty_table_detection
20 raise ValueError(f"Count of data_instance is 0: {data_instances}")
21 ValueError: Count of data_instance is 0: <fate_arch.computing.standalone._table.Table object at 0x7fa07f54d828>
沒有事先上傳所需的數(shù)據(jù)表
20210908
dsl 定義了整個模型的運(yùn)行流程
fate 啟動流程
bash init.sh init 一定要執(zhí)行
https://fate.readthedocs.io/en/latest/search.html?q=PSI&check_keywords=yes&area=default#
搜索各個專業(yè)名詞
https://fate.readthedocs.io/en/latest/_build_temp/python/fate_client/pipeline/component/README.html?highlight=component#component-list
components lis 組件列表
fate flow是訓(xùn)練相關(guān)，serving做預(yù)測
打不開 fateboard 就執(zhí)行一下
bash init.sh init ？
FATE-Flow Client 命令行使用說明https://fate.readthedocs.io/en/latest/_build_temp/python/fate_client/flow_client/README_zh.html?highlight=model%20version
No such file or directory federate
訓(xùn)練,預(yù)測,部署的 guest,host id 不一致導(dǎo)致的
沒上傳數(shù)據(jù)
單機(jī)fate運(yùn)行步驟
- 啟動init_env.sh
- 啟動fate_flow_server
- 跑具體的代碼
ImportError: cannot import name 'DispatcherMiddleware'
# path to data
# default fate installation path
DATA_BASE = "/data/projects/fate"
Toy_example Test
cd standalone_fate_master_${version}
source bin/init_env.sh
python ./examples/toy_example/run_toy_example.py 10000 10000 0(venv) [root@localhost standalone_fate_master_1.6.1]# 啟動環(huán)境之后 才能各種路徑識別20210902
venv 目錄在聯(lián)邦自己的路徑下
最主要是啟動這個server.py 文件 啟動服務(wù)器
https://gitee.com/WeBank/FATE/blob/master/doc/develop_guide_zh.rst
fate中文
https://blog.csdn.net/WenDong1997/article/details/106743620/
聯(lián)邦學(xué)習(xí)框架FATE實(shí)踐（訓(xùn)練/測試步驟及參數(shù)說明）
總結(jié)
以上是生活随笔為你收集整理的联邦学习fate笔记小结的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： pyspark性能调优参数
- 下一篇： typora使用手册