[QUESTION] torch broadcast error
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 20/100
- Issue 类型
- 缺陷
- 描述清晰度
- 需要澄清
- 活跃度
- 停滞
调研方向
Start from the traceback path: vescale/dtensor/redistribute.py _replicate_tensor, then mesh_broadcast in vescale/dtensor/_collective_utils.py, and compare the computed broadcast root against the device-mesh group size. Reproduce with examples/open_llama_4D_benchmark/run_open_llama_w_vescale.py and NCCL_DEBUG=WARN, and check whether the issue is torch compatibility or root-rank construction. Done means the broadcast root is valid for the group and the example no longer raises NCCL invalid-root.
由索引模型根据 Issue 内容生成。
描述
When I run run_open_llama_w_vescale.py with torch version 2.5.1+cu124, I met the following error:
[rank4]: Traceback (most recent call last):
[rank4]: File "/code/veScale/examples/open_llama_4D_benchmark/run_open_llama_w_vescale-ljx.py", line 104, in
[rank4]: vescale_model = parallelize_module(model, device_mesh["TP"], sharding_plan)
[rank4]: File "/code/veScale/vescale/dmodule/api.py", line 276, in parallelize_module
[rank4]: DModule.init_parameters(module, is_model_sharded)
[rank4]: File "/code/veScale/vescale/dmodule/_dmodule.py", line 302, in init_parameters
[rank4]: buffer = DModule._distribute_parameter(buffer, module._device_mesh, buffer_pi, is_sharded)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank4]: return func(*args, **kwargs)
[rank4]: File "/code/veScale/vescale/dmodule/_dmodule.py", line 266, in _distribute_parameter
[rank4]: dt = distribute_tensor(t, device_mesh, pi.placements)
[rank4]: File "/code/veScale/vescale/dtensor/api.py", line 252, in distribute_tensor
[rank4]: local_tensor = _replicate_tensor(local_tensor, device_mesh, idx)
[rank4]: File "/code/veScale/vescale/dtensor/redistribute.py", line 191, in _replicate_tensor
[rank4]: tensor = mesh_broadcast(tensor, mesh, mesh_dim=mesh_dim)
[rank4]: File "/code/veScale/vescale/dtensor/_collective_utils.py", line 273, in mesh_broadcast
[rank4]: aysnc_tensor = funcol.broadcast(tensor, src=src_for_dim, group=dim_group)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/distributed/_functional_collectives.py", line 153, in broadcast
[rank4]: tensor = torch.ops._c10d_functional.broadcast(self, src, group_name)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/_ops.py", line 1116, in call
[rank4]: return self._op(*args, **(kwargs or {}))
[rank4]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2777, invalid argument (run with NCCL_DEBUG=WARN for details), NCCL version 2.21.5
[rank4]: ncclInvalidArgument: Invalid value for an argument.
[rank4]: Last error:
[rank4]: Broadcast : invalid root 4 (root should be in the 0..4 range)
Is this because the torch version is not compatible?
- 主要语言
- Python
- 星标
- 1k
- 派生
- 65
- 平均合并
- 21 小时 34 分钟
- 30 天内合并 PR
- 1
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
volcengine/veScale 的其他 Issue
-
难度 1/5 1 小时以内 新手友好度 12/100
volcengine/veScale#75 ·
-
难度 5/5 一周以上 新手友好度 10/100
volcengine/veScale#73 · 1 条评论 · 3 个 reaction ·
-
难度 5/5 一周以上 新手友好度 8/100
volcengine/veScale#72 · 5 个 reaction ·
-
难度 3/5 1-2 天 新手友好度 30/100
volcengine/veScale#68 ·
-
难度 5/5 一周以上 新手友好度 8/100
volcengine/veScale#65 ·
查看 volcengine/veScale 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 66/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
docling-project/docling-pipelines#669 ·
维护者通常 1 天内回复
-
blocked kind/enhancement
难度 2/5 1-3 小时 新手友好度 78/100
opendatahub-io/agent-eval-harness#329 · 2 条评论 · 1 个 reaction ·
-
cc_wrapper crashes with ValueError on Windows when a path is on another drive可能已有人在做 @TrapsterDK 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 79/100
tweag/rules_haskell#2492 ·
维护者通常 1 天内回复