[QUESTION] torch broadcast error

未关闭
#62 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
20/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
停滞
技术栈
python, pytorch

调研方向

Start from the traceback path: vescale/dtensor/redistribute.py _replicate_tensor, then mesh_broadcast in vescale/dtensor/_collective_utils.py, and compare the computed broadcast root against the device-mesh group size. Reproduce with examples/open_llama_4D_benchmark/run_open_llama_w_vescale.py and NCCL_DEBUG=WARN, and check whether the issue is torch compatibility or root-rank construction. Done means the broadcast root is valid for the group and the example no longer raises NCCL invalid-root.

由索引模型根据 Issue 内容生成。

描述

When I run run_open_llama_w_vescale.py with torch version 2.5.1+cu124, I met the following error:

[rank4]: Traceback (most recent call last):
[rank4]: File "/code/veScale/examples/open_llama_4D_benchmark/run_open_llama_w_vescale-ljx.py", line 104, in
[rank4]: vescale_model = parallelize_module(model, device_mesh["TP"], sharding_plan)
[rank4]: File "/code/veScale/vescale/dmodule/api.py", line 276, in parallelize_module
[rank4]: DModule.init_parameters(module, is_model_sharded)
[rank4]: File "/code/veScale/vescale/dmodule/_dmodule.py", line 302, in init_parameters
[rank4]: buffer = DModule._distribute_parameter(buffer, module._device_mesh, buffer_pi, is_sharded)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank4]: return func(*args, **kwargs)
[rank4]: File "/code/veScale/vescale/dmodule/_dmodule.py", line 266, in _distribute_parameter
[rank4]: dt = distribute_tensor(t, device_mesh, pi.placements)
[rank4]: File "/code/veScale/vescale/dtensor/api.py", line 252, in distribute_tensor
[rank4]: local_tensor = _replicate_tensor(local_tensor, device_mesh, idx)
[rank4]: File "/code/veScale/vescale/dtensor/redistribute.py", line 191, in _replicate_tensor
[rank4]: tensor = mesh_broadcast(tensor, mesh, mesh_dim=mesh_dim)
[rank4]: File "/code/veScale/vescale/dtensor/_collective_utils.py", line 273, in mesh_broadcast
[rank4]: aysnc_tensor = funcol.broadcast(tensor, src=src_for_dim, group=dim_group)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/distributed/_functional_collectives.py", line 153, in broadcast
[rank4]: tensor = torch.ops._c10d_functional.broadcast(self, src, group_name)
[rank4]: File "/miniconda3-new/envs/llm-cuda12.4-vescale/lib/python3.10/site-packages/torch/_ops.py", line 1116, in call
[rank4]: return self._op(*args, **(kwargs or {}))
[rank4]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2777, invalid argument (run with NCCL_DEBUG=WARN for details), NCCL version 2.21.5
[rank4]: ncclInvalidArgument: Invalid value for an argument.
[rank4]: Last error:
[rank4]: Broadcast : invalid root 4 (root should be in the 0..4 range)

Is this because the torch version is not compatible?

主要语言
Python
星标
1k
派生
65
平均合并
21 小时 34 分钟
30 天内合并 PR
1

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

volcengine/veScale 的其他 Issue

查看 volcengine/veScale 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。