bughelp wanted
仓库指标
- 星标
- (1,513 个星标)
- PR 合并指标
- (平均合并 9天 17小时) (30 天内合并 200 个 PR)
描述
A reported bug from the community that the latest Miles Docker can not run Qwen3 235B MoE models. I highly think this might be wrong, but the errors are:
[36m(MegatronTrainRayActor pid=85329, ip=10.142.198.76)[0m [rank51]:[E129 09:24:37.913960277 ProcessGroupNCCL.cpp:683] [Rank 51] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=25, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600055 milliseconds before timing out.
An all reduce timeout. I think it's a good start point for those feel interested in Large scale training to debug on.