[BUG] cuOpt server crashes (SIGBUS) on any GPU memory touch on NVIDIA DGX Spark (GB10 Superchip, compute capability 12.1)

Đang mở
  • Có gì không ổn?
#1,995 4 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@hlinsen đang làm issue này rồi.

Từ ngày 1/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

bug

Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.

Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"

exit code 135 (SIGBUS), no Python traceback

Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"

exit code 135 (SIGBUS)

Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).

Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.

Expected behavior

The solve request should complete normally, as it does on GPUs with more established compute capabilities.

Environment details

  • Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
  • Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
  • OS: Ubuntu 24.04.4 LTS, aarch64
  • Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
  • Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)

Additional context

PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.

No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.

Happy to provide the full SSH-reproducible command sequence if useful for triage.

Ngôn ngữ chính
Cuda
Star
1.1k
Fork
255
Merge trung bình
1 ngày 22 giờ
Pull request đã merge (30 ngày)
102

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/cuopt

Tất cả issue của NVIDIA/cuopt

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.