[BUG] cuOpt server crashes (SIGBUS) on any GPU memory touch on NVIDIA DGX Spark (GB10 Superchip, compute capability 12.1)

未关闭
#1,995 4 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@hlinsen 已经在做这个了。

开始于 2026年10月1日。

评估

这个 Issue 还没有评估数据。

描述

bug

Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.

Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"

exit code 135 (SIGBUS), no Python traceback

Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"

exit code 135 (SIGBUS)

Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).

Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.

Expected behavior

The solve request should complete normally, as it does on GPUs with more established compute capabilities.

Environment details

  • Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
  • Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
  • OS: Ubuntu 24.04.4 LTS, aarch64
  • Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
  • Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)

Additional context

PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.

No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.

Happy to provide the full SSH-reproducible command sequence if useful for triage.

主要语言
Cuda
星标
1.1k
派生
249
平均合并
1 天 15 小时
30 天内合并 PR
126

环境准备

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

NVIDIA/cuopt 的其他 Issue

查看 NVIDIA/cuopt 的全部 Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。