[BUG]: multi-GPU cuda.core samples take the CUDA device count from NVML and break under CUDA_VISIBLE_DEVICES
维护者通常 1 天内回复
评估
- 难度
- 2/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 78/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- python
调研方向
阅读 samples/cuda_core/simpleP2P/simpleP2P.py、simpleMultiGpu 和 multiGPUGradientAverage，然后将它们使用的设备数量与 Device.get_all_devices() 和 systemInfo 进行比较。更新 issue 中描述的三个示例和 README 的两行；当示例中的子集可见 GPU 数量正常工作，且 systemInfo 保留 NVML 的系统范围数量时，任务即告完成。issue 报告称，由于缺少 mpi4py，multiGPUGradientAverage 未经验证。
由索引模型根据 Issue 内容生成。
描述
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
samples/cuda_core/simpleP2P, simpleMultiGpu and multiGPUGradientAverage size their device lists with cuda.core.system.get_num_devices(). That function returns nvml.device_get_count_v2(), which counts every GPU on the system and ignores CUDA_VISIBLE_DEVICES. On an 8-GPU B300 host with two GPUs visible, simpleP2P reports 8 devices and then fails on Device(2):
CUDA-capable device count: 8
ValueError: device_id must be within [0, 2), got 2
The other two samples are affected the same way:
multiGPUGradientAveragemapsrank % num_gpus, so it can pick a device that is not visible.simpleMultiGpuonly usesDevice(0)andDevice(1), so it survives, but its README describessystem.get_num_devices()as the "count of visible CUDA devices", which it is not.
This turns up whenever a job gets a subset of a node's GPUs: shared clusters, schedulers, or CI runners that pin devices. The nightly rows are single-GPU, so they skip these samples.
How to Reproduce
CUDA_VISIBLE_DEVICES=<two of the GPUs> python samples/cuda_core/simpleP2P/simpleP2P.py
Expected behavior
The samples count the CUDA devices the process can use (len(Device.get_all_devices())), so they work when CUDA_VISIBLE_DEVICES selects a subset of the GPUs.
Proposed fix
Count CUDA devices with len(Device.get_all_devices()) in the three samples and fix the two README lines. system.get_num_devices() stays the right call in systemInfo, which reports NVML's view. I have the change ready. With two of the eight GPUs visible:
- Before:
simpleP2Pfails andsimpleMultiGpupasses. - After: both pass.
multiGPUGradientAverage was skipped in both runs because mpi4py was missing, so it is not verified.
The fix was written with Claude Opus 5.5 and reviewed by the contributor.
Environment
GPU: B300 (cc 10.3, x86_64, 8 GPUs, 2 visible) | driver 610.57.04 | CUDA 13.4.2 (local CTK) | Python 3.12 | cuda-python 2da9c65a
- 主要语言
- Cython
- 星标
- 3.4k
- 派生
- 341
- 平均合并
- 1 天 18 小时
- 30 天内合并 PR
- 147
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
NVIDIA/cuda-python 的其他 Issue
-
CI/CD P1
难度 2/5 1-3 小时 新手友好度 78/100
NVIDIA/cuda-python#3273 · 1 条评论 ·
维护者通常 1 天内回复
-
[BUG]: cuda.bindings GL interop tests open EGL device 0 on headless multi-GPU hosts and fail with cudaErrorInvalidDevice可能已有人在做 @0z5a 于 1 天前认领。 未关闭triage
难度 2/5 1-3 小时 新手友好度 78/100
NVIDIA/cuda-python#3212 ·
维护者通常 1 天内回复
-
lychee should specify which links are failing可能已有人在做 @kkonnerharris 于 2 天前认领。 未关闭triage
难度 2/5 1-3 小时 新手友好度 67/100
NVIDIA/cuda-python#3204 · 2 条评论 · 已指派 1 人 ·
维护者通常 1 天内回复
-
[BUG]: cleanup-pr-previews never removes previews of deleted PRs可能已有人在做 @rwgk 于 22 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 84/100
NVIDIA/cuda-python#2952 · 已指派 1 人 ·
维护者通常 1 天内回复
-
[BUG]: LocatedHeaderDir is mutable, so callers can poison the cached header-directory lookup可能已有人在做 @rwgk 于 37 天前认领。 未关闭triage
难度 2/5 1-3 小时 新手友好度 82/100
NVIDIA/cuda-python#2646 · 1 个 reaction ·
维护者通常 1 天内回复