[BUG]: Program.compile("ptx") fails on machines without a CUDA driver, although NVRTC does not need one
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
Research direction
Start in cuda/core/_program.pyx at _can_load_generated_ptx() (around line 815) and the cache-hit branch of Program.compile (~line 1192); the raising call is driver_version() in cuda/core/_utils/version.pyx, which needs libcuda. Catch the DynamicLibNotFoundError and skip the loadability warning instead of failing, mirroring the cubin path. Reproduce with the snippet on a driverless machine and check existing cuda.core program tests; done means PTX compiles without a driver and a warning (or silent skip) replaces the error.
Written by the indexing model from the issue text.
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
On a Linux machine with no NVIDIA driver installed (a GitHub Actions ubuntu runner), compiling a Program to PTX with the NVRTC backend raises DynamicLibNotFoundError for libcuda.so.1. Compiling the same source to cubin with an explicit arch works on the same machine, so NVRTC itself is available.
The failure comes from the loadability check that runs before PTX compilation. _can_load_generated_ptx() calls driver_version(), which calls cuDriverGetVersion and needs libcuda. The check only decides whether to emit a RuntimeWarning, but when the driver library is missing it raises instead, and the whole compile fails. The same check is on the cache hit path in Program.compile.
This matters for CI jobs that compile kernels to PTX without a GPU, for example to inspect the generated PTX in tests. My workaround was to call cuda.bindings.nvrtc directly with the flags from ProgramOptions.as_bytes("nvrtc", "ptx"), which works without a driver.
How to Reproduce
On a machine without an NVIDIA driver. I used the ubuntu latest runner on GitHub Actions with Python 3.12.
pip install "cuda-core[cu12]==1.2.1"
from cuda.core import Program, ProgramOptions
src = 'extern "C" __global__ void k(float* x) { x[0] = 2.0f * x[0] + 1.0f; }'
# Works without a driver
Program(src, code_type="c++", options=ProgramOptions(arch="sm_75")).compile("cubin")
# Fails without a driver
Program(src, code_type="c++", options=ProgramOptions(arch="compute_75")).compile("ptx")
A public run on a runner with no libcuda shows both cases, the cubin compile passes and the ptx compile fails. https://github.com/VolodymyrLinuxovich/heat-risk-cuda/actions/runs/37217831804
Trimmed traceback from the failing run
cuda/core/_program.pyx:222: in cuda.core._program.Program.compile
cuda/core/_program.pyx:738: in cuda.core._program._program_compile_uncached
cuda/core/_program.pyx:1192: in cuda.core._program.Program_compile
cuda/core/_program.pyx:815: in cuda.core._program._can_load_generated_ptx
cuda/core/_utils/version.pyx:37: in cuda.core._utils.version.driver_version
cuda/bindings/driver.pyx:23154: in cuda.bindings.driver.cuDriverGetVersion
...
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: "cuda" is an NVIDIA driver library and can only be found via system search. Ensure the NVIDIA display driver is installed.
The _can_load_generated_ptx code on main looks the same as in 1.2.1, so I expect main behaves the same way, but I have only run 1.2.1.
Expected behavior
PTX compilation succeeds without a driver, the same as cubin compilation. When the driver version cannot be determined, the loadability check could skip the warning (or warn that loadability is unknown) instead of failing the compile.
I would be glad to open a PR for this if that approach sounds right to you.
Operating System
Ubuntu (GitHub Actions ubuntu latest runner), no NVIDIA driver
nvidia-smi output
Not available, no driver on that machine.
- Dominant language
- Cython
- Stars
- 3.4k
- Forks
- 334
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 123
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA/cuda-python
-
triage
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
NVIDIA/cuda-python#2952 ·
Maintainers usually reply within 1 day
-
[DOC]: cuda.core 1.1.1 note misstates program cache permissionsPossibly taken @leofang claimed this 11 days ago. Opendocumentation P1
Difficulty 1/5 Under an hour Newbie friendliness 88/100
NVIDIA/cuda-python#2717 · 1 assignee ·
Maintainers usually reply within 1 day
-
[DOC]: `PinnedMemoryResource.allocate` documents no parametersPossibly taken @Andy-Jost claimed this 11 days ago. Opencuda.core documentation P1
Difficulty 1/5 1-3 hours Newbie friendliness 90/100
NVIDIA/cuda-python#2712 · 1 assignee ·
Maintainers usually reply within 1 day
-
[BUG]: LocatedHeaderDir is mutable, so callers can poison the cached header-directory lookupPossibly taken @rwgk claimed this 11 days ago. Opentriage
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
NVIDIA/cuda-python#2646 · 1 reaction ·
Maintainers usually reply within 1 day
-
[FEA]: Support inheritance from BufferPossibly taken @leofang claimed this 11 days ago. Opencuda.core triage
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
NVIDIA/cuda-python#2435 · 1 comment ·
Maintainers usually reply within 1 day
All issues in NVIDIA/cuda-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
rubys/roundhouse#444 ·
Maintainers usually reply within 1 day
-
vxc prints a debug line '[flat-codegen] emitted module via the flat path' on every compilePossibly taken @YodHeVauHe claimed this today. Opendevex good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
kmmbvnr/rank#196 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
SciML/ModelingToolkit.jl#5255 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
NVIDIA/cuda-quantum#5539 ·
Maintainers usually reply within 1 day