[BUG] cuOpt server crashes (SIGBUS) on any GPU memory touch on NVIDIA DGX Spark (GB10 Superchip, compute capability 12.1)

Abierto
  • ¿Algo no está bien?
#1,995 4 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@hlinsen ya está trabajando en esto.

Desde el 1/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

bug

Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.

Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"

exit code 135 (SIGBUS), no Python traceback

Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"

exit code 135 (SIGBUS)

Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).

Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.

Expected behavior

The solve request should complete normally, as it does on GPUs with more established compute capabilities.

Environment details

  • Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
  • Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
  • OS: Ubuntu 24.04.4 LTS, aarch64
  • Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
  • Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)

Additional context

PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.

No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.

Happy to provide the full SSH-reproducible command sequence if useful for triage.

Lenguaje dominante
Cuda
Estrellas
1.1k
Forks
249
Merge medio
1 d 11 h
PR fusionados (30 d)
123

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/cuopt

Todos los issues de NVIDIA/cuopt

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.