[BUG]: multi-GPU cuda.core samples take the CUDA device count from NVML and break under CUDA_VISIBLE_DEVICES
維護者通常 1 天內回覆
評估
- 難度
- 2/5
- 預估耗時
- 1-3 小時
- 新手友好度
- 78/100
- Issue 類型
- 缺陷
- 描述清晰度
- 描述清楚
- 活躍度
- 活躍
- 技術堆疊
- python
研究方向
閱讀 samples/cuda_core/simpleP2P/simpleP2P.py、simpleMultiGpu 和 multiGPUGradientAverage，然後將它們使用的裝置數量與 Device.get_all_devices() 和 systemInfo 進行比較。更新 issue 中描述的三個範例和 README 的兩行；當範例中的子集合可見 GPU 數量正常運作，且 systemInfo 保留 NVML 的系統範圍數量時，任務即告完成。issue 報告指出，由於缺少 mpi4py，multiGPUGradientAverage 未經驗證。
由索引模型根據 Issue 內容生成。
描述
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
samples/cuda_core/simpleP2P, simpleMultiGpu and multiGPUGradientAverage size their device lists with cuda.core.system.get_num_devices(). That function returns nvml.device_get_count_v2(), which counts every GPU on the system and ignores CUDA_VISIBLE_DEVICES. On an 8-GPU B300 host with two GPUs visible, simpleP2P reports 8 devices and then fails on Device(2):
CUDA-capable device count: 8
ValueError: device_id must be within [0, 2), got 2
The other two samples are affected the same way:
multiGPUGradientAveragemapsrank % num_gpus, so it can pick a device that is not visible.simpleMultiGpuonly usesDevice(0)andDevice(1), so it survives, but its README describessystem.get_num_devices()as the "count of visible CUDA devices", which it is not.
This turns up whenever a job gets a subset of a node's GPUs: shared clusters, schedulers, or CI runners that pin devices. The nightly rows are single-GPU, so they skip these samples.
How to Reproduce
CUDA_VISIBLE_DEVICES=<two of the GPUs> python samples/cuda_core/simpleP2P/simpleP2P.py
Expected behavior
The samples count the CUDA devices the process can use (len(Device.get_all_devices())), so they work when CUDA_VISIBLE_DEVICES selects a subset of the GPUs.
Proposed fix
Count CUDA devices with len(Device.get_all_devices()) in the three samples and fix the two README lines. system.get_num_devices() stays the right call in systemInfo, which reports NVML's view. I have the change ready. With two of the eight GPUs visible:
- Before:
simpleP2Pfails andsimpleMultiGpupasses. - After: both pass.
multiGPUGradientAverage was skipped in both runs because mpi4py was missing, so it is not verified.
The fix was written with Claude Opus 5.5 and reviewed by the contributor.
Environment
GPU: B300 (cc 10.3, x86_64, 8 GPUs, 2 visible) | driver 610.57.04 | CUDA 13.4.2 (local CTK) | Python 3.12 | cuda-python 2da9c65a
- 主要語言
- Cython
- 星號
- 3.4k
- 分支
- 341
- 平均合併
- 1 天 18 小時
- 30 天內合併 PR
- 148
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
NVIDIA/cuda-python 的其他 Issue
-
CI/CD P1
難度 2/5 1-3 小時 新手友好度 78/100
NVIDIA/cuda-python#3273 · 1 則留言 ·
維護者通常 1 天內回覆
-
[BUG]: cuda.bindings GL interop tests open EGL device 0 on headless multi-GPU hosts and fail with cudaErrorInvalidDevice可能已有人在做 @0z5a 於 1 天前認領。 未關閉triage
難度 2/5 1-3 小時 新手友好度 78/100
NVIDIA/cuda-python#3212 ·
維護者通常 1 天內回覆
-
lychee should specify which links are failing可能已有人在做 @kkonnerharris 於 2 天前認領。 未關閉triage
難度 2/5 1-3 小時 新手友好度 67/100
NVIDIA/cuda-python#3204 · 2 則留言 · 已指派 1 人 ·
維護者通常 1 天內回覆
-
[BUG]: cleanup-pr-previews never removes previews of deleted PRs可能已有人在做 @rwgk 於 22 天前認領。 未關閉
難度 2/5 1-3 小時 新手友好度 84/100
NVIDIA/cuda-python#2952 · 已指派 1 人 ·
維護者通常 1 天內回覆
-
[DOC]: cuda.core 1.1.1 note misstates program cache permissions可能已有人在做 @leofang 於 36 天前認領。 未關閉documentation P1
難度 1/5 1 小時以內 新手友好度 88/100
NVIDIA/cuda-python#2717 · 已指派 1 人 ·
維護者通常 1 天內回覆