SimAI_simulator不输出结果，bug修复求助
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Domain
- distributed-systems
Research direction
Start by reproducing the command using scripts/megatron_workload_with_aiob.sh, the generated workload file, bin/SimAI_simulator, and astra-sim-alibabacloud/inputs/config/SimAI.conf. Compare its behavior with the repository workload example and inspect where the simulator reads or processes the larger workload. Done means the provided workload produces simulation output under the documented environment.
Written by the indexing model from the issue text.
Description
使用aicb生成了workload，执行命令如下
sh ./scripts/megatron_workload_with_aiob.sh -m 7 \ --world_size 128 --tp 8 --pp 1 \ --frame Megatron --gpu_type H800 \ --global_batch 1024 --micro_batch 8 \ --seq_length 4096 --swiglu \ --use_flash_attn --aiob_enable
生成的workload是没有问题的。
运行./bin/SimAI_analytical可以输出结果，没有问题。
但是simulation仿真没有结果输出，我的执行环境也是和仓库一致的。GCC/G++ 9.4.0, python 3.8.10 in Ubuntu 20.04
AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/H800-gpt_7B-world_size128-tp8-pp1-ep1-gbs1024-mbs8-seq4096-MOE-False-GEMM-False-flash_attn-True.txt \ -n ./Spectrum-X_128g_8gps_400Gbps_H800 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
如果用仓库给的workload例子，可以输出结果
HYBRID_TRANSFORMER_FWD_IN_BCKWD model_parallel_NPU_group: 8 ep: 1 pp: 1 vpp: 8 ga: 1 all_gpus: 32 checkpoints: 0 checkpoint_initiates: 0 2 embedding_layer -1 556000 ALLREDUCE 16777216 1 NONE 0 1 NONE 0 1 embedding_layer -1 556000 ALLREDUCE 67108864 1
但是这个workload过于简单，不是一个完整的模型。
所以代码应该是有bug的，这个问题在很多issue中已经提到，并未解决。你们有可能解决这个bug，并更新代码仓库吗？
公司领导现在想要用这个仓库做很多工作，目前simai不是很work。
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 6h 47m
- Merged PRs (30d)
- 2
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from aliyun/SimAI
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
SimCCL代码 Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100