[BUG] cuOpt server crashes (SIGBUS) on any GPU memory touch on NVIDIA DGX Spark (GB10 Superchip, compute capability 12.1)
维护者通常 1 天内回复
@hlinsen 已经在做这个了。
开始于 2026年10月1日。
评估
这个 Issue 还没有评估数据。
描述
Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.
Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"
exit code 135 (SIGBUS), no Python traceback
Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:
docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"
exit code 135 (SIGBUS)
Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).
Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.
Expected behavior
The solve request should complete normally, as it does on GPUs with more established compute capabilities.
Environment details
- Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
- Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
- OS: Ubuntu 24.04.4 LTS, aarch64
- Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
- Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)
Additional context
PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.
No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.
Happy to provide the full SSH-reproducible command sequence if useful for triage.
- 主要语言
- Cuda
- 星标
- 1.1k
- 派生
- 249
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 126
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/cuopt 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 82/100
维护者通常 1 天内回复
-
难度 4/5 3-5 天 新手友好度 32/100
维护者通常 1 天内回复
-
CI: nightly Cython build with bounds checking enabled可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 4/5 3-5 天 新手友好度 22/100
NVIDIA/cuopt#2255 · 1 个 reaction ·
维护者通常 1 天内回复
-
难度 3/5 半天 新手友好度 50/100
维护者通常 1 天内回复