bughelp wanted
倉庫指標
- 星標
- (1,513 顆星)
- PR 合併指標
- (平均合併 9天 17小時) (30 天內合併 200 個 PR)
描述
A reported bug from the community that the latest Miles Docker can not run Qwen3 235B MoE models. I highly think this might be wrong, but the errors are:
[36m(MegatronTrainRayActor pid=85329, ip=10.142.198.76)[0m [rank51]:[E129 09:24:37.913960277 ProcessGroupNCCL.cpp:683] [Rank 51] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=25, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600055 milliseconds before timing out.
An all reduce timeout. I think it's a good start point for those feel interested in Large scale training to debug on.