camera_viz setup fails with a distro CUDA toolkit: "Cannot determine the CUDA toolkit version at /usr"

Start with examples/camera_viz/scripts/check_cuda.py, _install_deps.sh, and camera_viz.sh; then read tests/python/examples/camera_viz/test_cuda_setup.py and the CUDA lookup notes in docs/source/references/camera_streaming.rst. Reproduce the distro layout and PATH ordering cases described in the issue, then check how the detected root reaches CuPy during setup and viewer runs. Done means detection and runtime checks work for the reported layouts and the tests and docs cover the behavior.

Summary

examples/camera_viz/camera_viz.sh setup aborts on machines where CUDA comes from the Ubuntu/Debian nvidia-cuda-toolkit package. The toolkit works: the same machine builds Isaac Capture from source with BUILD_VIZ=ON, and build_from_source/index.rst lists nvidia-cuda-toolkit as a supported way to get it. Fixing version detection alone is not enough. With CuPy 14, the viewer also needs CUDA_PATH=/usr at runtime to find the headers.

=== Local setup ===
CUDA check failed: Cannot determine the CUDA toolkit version at /usr. Install the toolkit matching your JetPack/driver and set CUDA_PATH to its directory, then rerun setup.
error: CUDA toolkit detection failed

Environment

• Ubuntu 24.04, x86_64, RTX 6000 Ada, driver 580
• nvidia-cuda-toolkit 12.0.140 from the Ubuntu archive; no /usr/local/cuda*; CUDA_PATH unset
• /usr/bin/nvcc is a #!/bin/sh wrapper around /usr/lib/nvidia-cuda-toolkit/bin/nvcc
• Headers are in /usr/include, and libnvrtc.so.12 is in /usr/lib/x86_64-linux-gnu
• main at 1339a9cd3

Steps to reproduce

Detection alone (system Python, the way setup calls it):

python3 examples/camera_viz/scripts/check_cuda.py --detect
# CUDA check failed: Cannot determine the CUDA toolkit version at /usr. ...
CUDA_PATH=/usr python3 examples/camera_viz/scripts/check_cuda.py --detect
# same error: no CUDA_PATH value is accepted for this layout

The kernel check that setup runs last, in a venv with the CuPy that setup installs:

uv venv /tmp/cupy-venv && uv pip install --python /tmp/cupy-venv/bin/python cupy-cuda12x numpy
/tmp/cupy-venv/bin/python examples/camera_viz/scripts/check_cuda.py --cuda-major 12
# CUDA check failed: Failed to find CUDA headers. ... specify CUDA_PATH environment variable.
CUDA_PATH=/usr /tmp/cupy-venv/bin/python examples/camera_viz/scripts/check_cuda.py --cuda-major 12
# CuPy kernel check passed: NVIDIA RTX 6000 Ada Generation, CuPy 14.2.0, NVRTC 12.0

Root cause

There are two independent gaps. Each one blocks this layout on its own.

1. Setup cannot read the toolkit version. detect_toolkit() picks a root from CUDA_PATH, else dirname(dirname(nvcc)) for the first nvcc on PATH, else /usr/local/cuda. It then reads the version only from <root>/version.json, <root>/version.txt or a cuda-X.Y directory name. The distro layout roots at /usr and has none of the three. _install_deps.sh:92 treats the error as fatal.

Two more consequences of the same logic:

• Only the first nvcc on PATH is considered, with no fallback. A machine with an official toolkit and a distro nvcc earlier on PATH fails the same way.
• The pip nvidia-cuda-nvcc-cu12 wheel ships no nvcc binary. A machine whose CUDA comes only from pip wheels falls through to /usr/local/cuda and fails too, even though CuPy can run from those wheels.

CMake's FindCUDAToolkit is not affected, because it gets the version by running nvcc --version.

2. CuPy 14 does not find distro headers on its own. Setup installs an unpinned cupy-cuda12x, which is currently 14.2.0. CuPy 14.2.0 locates headers through cuda.pathfinder (1.8.4 here), which searches, in order: NVIDIA wheels in site-packages, conda, CUDA_PATH/CUDA_HOME, a root derived from the loaded libcudart, and fixed system directories. On this layout only CUDA_PATH=/usr resolves to /usr/include. CuPy 13.6 derived its root from nvcc on PATH and passes the same kernel check without CUDA_PATH. Nothing in setup or camera_viz.sh exports CUDA_PATH. So even with gap 1 fixed, the final setup check, camera_viz.sh run and the camera_viz GPU tests all fail with "Failed to find CUDA headers".

Regression

Setup started failing with #1194 (bb59d39b6). Before it, setup defaulted to CUDA 12.0 unless /usr/local/cuda pointed at a cuda-X.Y directory. On this machine that picked cupy-cuda12x and finished. Whether the viewer then worked depended on the CuPy release resolved: 13.6.0 runs kernels on this layout without CUDA_PATH, and 14.2.0 does not. #1194 moved that runtime failure into setup, and its detection rejects CUDA_PATH=/usr, which is the value CuPy 14 needs.

Who is affected

• Distro nvidia-cuda-toolkit installs, including users who followed build_from_source/index.rst. setup --build-from-source points at that same page, but detection fails before setup reaches the build.
• Machines where a distro nvcc precedes an official toolkit on PATH.
• Machines whose CUDA comes only from pip wheels.
• Every setup mode is affected, --sender-only included, because detection runs unconditionally.
• Not affected, by inspection of the logic: the official runfile/deb installers and JetPack (/usr/local/cuda-X.Y with a version file). Conda layouts were not checked.

Suggested fix

1. Detection. When no version file is present, parse nvcc --version, as CMake does. Treat a distro nvcc as rooted at /usr once /usr/include/cuda_runtime.h exists. Try each nvcc on PATH, then /usr/local/cuda, and use the first one that resolves, rather than only the first nvcc found.
2. Runtime. Export the detected root as CUDA_PATH for setup's kernel check and for camera_viz.sh run/loopback, so setup verifies the environment the viewer will use. One way is to write it next to the venv and source it in camera_viz.sh; setting it in the generated .venv/bin/activate would cover users who activate the venv.
   • Alternative: install cupy-cuda12x[ctk] so CuPy takes headers and NVRTC from wheels. This decouples CuPy from the system layout, but makes the venv much larger and needs checking on Jetson.
3. Messages and docs. Drop "JetPack" from the desktop error, or make it conditional, and say which layouts are recognized. Update camera_streaming.rst:82-84, which currently describes setup's lookup order but not the layouts it accepts or what CuPy needs at runtime.
4. Tests. Extend tests/python/examples/camera_viz/test_cuda_setup.py with a distro layout (an nvcc wrapper under a temporary usr/bin, nvcc --version stubbed) and a PATH ordering case.

Workaround

Give setup a cuda-X.Y directory to read the version from, and give CuPy headers through it. The name must match the installed toolkit:

mkdir -p ~/.local/cuda-12.0 && ln -s /usr/include ~/.local/cuda-12.0/include
export CUDA_PATH=~/.local/cuda-12.0   # keep it set for setup and for every run

With this, check_cuda.py --detect and check_cuda.py --cuda-major 12 both pass on the machine above. A full setup run with the workaround has not been tested.