bughelp wanted
Repository metrics
- Stars
- (1,513 個のスター)
- PR merge metrics
- (平均マージ 9d 17h) (30d で 200 merged PRs)
説明
A reported bug from the community that the latest Miles Docker can not run Qwen3 235B MoE models. I highly think this might be wrong, but the errors are:
[36m(MegatronTrainRayActor pid=85329, ip=10.142.198.76)[0m [rank51]:[E129 09:24:37.913960277 ProcessGroupNCCL.cpp:683] [Rank 51] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=25, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600055 milliseconds before timing out.
An all reduce timeout. I think it's a good start point for those feel interested in Large scale training to debug on.