Dev add bert automated test #203
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
util.py Add GPU memory for printing. analysis.py Analyze log files to get loss and GPU memory. result_analysis.py Compare the two results to generate a png image. stitching_pic.py Stitching pictures. train_perbert.sh Training script. train_perbert_list.sh Training list.
This reverts commit 647dbd6.
util.py Add GPU memory for printing. analysis.py Analyze log files to get loss and GPU memory. result_analysis.py Compare the two results to generate a png image. stitching_pic.py Stitching pictures. train_perbert.sh Training script. train_perbert_list.sh Training list.
luqiang-guo
added
documentation
|
能不能把那些bash脚本改成python的，复杂的bash脚本很难维护 |
|
这个测试能做到无人值守吗，就是脚本自己能判断精度和性能是否达标 |
可以做到自己判断精度，而且还可以出一个分析报告。 |
之前试着改过python的方式，但是尝试失败了。主要问题是我需要非阻塞的在其他机上执行启动脚本的命令。 |
用 subprocess 模块的poll 或者 async await 都可以做到在python里面的非阻塞 |
好的，我后面用async await 修改 |
luqiang-guo
added
documentation
ouyangyu
reviewed
Jun 22, 2021
| if __name__ == "__main__": | ||
| main() | ||
|
|
||
|
|
| PRINT_ITER=1 | ||
| ITER_NUM=130 | ||
|
|
||
| NODE_IPS='10.10.0.2','10.10.0.3','10.10.0.4','10.10.0.5' |
|
|
||
| NODE_IPS='10.10.0.2','10.10.0.3','10.10.0.4','10.10.0.5' | ||
|
|
||
| # INIT_MODEL=/opt/initial_model |
|
|
||
| # ############################################################################### | ||
| # # f32 adam | ||
| # ############################################################################### |
| @@ -0,0 +1,134 @@ | |||
| import json | |||
| import matplotlib.pyplot as plt | |||
| PYTHON="python3.8" | ||
| DOCKER_USER=root | ||
|
|
||
| multi_machine() |
| # param 5 | ||
| IS_F32=$5 | ||
|
|
||
| declare -a host_list=("10.10.0.2" "10.10.0.3" "10.10.0.4" "10.10.0.5") |
| ######################################################################################## | ||
| ALL_NODES=4 | ||
|
|
||
| declare -a host_list=("10.11.0.2" "10.11.0.3" "10.11.0.4" "10.11.0.5") |
|
|
||
| ####################################################################################### | ||
| # 2 run single | ||
| ######################################################################################## |
LanguageModeling/BERT/run_squad.sh
Outdated
| @@ -0,0 +1,74 @@ | |||
| BENCH_ROOT_DIR=/home/oyy/workspace/OneFlow-Benchmark/LanguageModeling/BERT | |||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add docker image multi-machine startup script and bert multi-machine execution script.