[BUG] cuOpt server crashes (SIGBUS) on any GPU memory touch on NVIDIA DGX Spark (GB10 Superchip, compute capability 12.1)

オープン
#1,995 コメント 4 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@hlinsen がすでに取り組んでいます。

2026年10月1日 から。

評価

この issue はまだ評価されていません。

説明

bug

Running the cuopt-server container on an NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, compute capability 12.1), the FastAPI layer starts fine and /cuopt/health responds {"status":"RUNNING"}, but any submitted POST /cuopt/request never completes. The background solver worker pool (process_handler.create_process) crash-loops continuously — each forked worker dies immediately and is respawned every ~0.4-0.5s, forever, with no application-level error ever logged.

Isolated the crash to plain GPU memory allocation via RAPIDS, independent of any cuOpt-specific code:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import cupy as cp; print(cp.arange(10).sum())"

exit code 135 (SIGBUS), no Python traceback

Also reproduces with numba.cuda (no RMM/cuDF involved at all), and crashes even earlier — on device capability query, before any kernel is compiled or launched:

docker run --rm --gpus all nvidia/cuopt:latest-cu13
python -c "import numba.cuda as cuda; print(cuda.is_available())"

exit code 135 (SIGBUS)

Confirmed this is not a stale-package issue: reproduces identically on today's newest available nightly (nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64, bundling cupy-cuda13x 14.2.0 and rmm-cu13 26.10.0a44 — the latest published anywhere).

Also confirmed it's not about RMM's allocator strategy — CudaMemoryResource, CudaAsyncMemoryResource, and ManagedMemoryResource all crash identically.

Expected behavior

The solve request should complete normally, as it does on GPUs with more established compute capabilities.

Environment details

  • Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), compute_cap=12.1
  • Driver: 580.173.02, CUDA 13.0 (per nvidia-smi)
  • OS: Ubuntu 24.04.4 LTS, aarch64
  • Docker 29.2.1, NVIDIA Container Toolkit 1.19.1
  • Images tested: nvidia/cuopt:latest-cu13 (26.8.0 stable) and nvidia/cuopt:26.10.0a-cuda13.3-py3.14-arm64 (nightly)

Additional context

PyTorch works correctly on this same GPU (confirmed via a container already running vllm-openai on this box): it prints a compute-capability-mismatch warning (Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)) but still runs GPU computations correctly. This suggests the underlying driver/hardware is fine, and the issue is specific to how RAPIDS' CUDA-driver-API bindings (used internally by libcuopt/librmm) handle an unrecognized/newer compute-capability value — the crash occurs at device-query time, before any kernel-image compatibility issue would normally surface, which points to a client-side bug (e.g. an unbounded architecture lookup) rather than a missing compiled kernel.

No kernel-level fault (NVRM: Xid) is logged at the time of the crash (journalctl -k), confirming this is a userspace crash, not a GPU-reported hardware fault.

Happy to provide the full SSH-reproducible command sequence if useful for triage.

主要言語
Cuda
スター
1.1k
フォーク
255
平均マージ
1日 11時間
マージ済み PR(30日)
123

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/cuopt のほかの issue

NVIDIA/cuopt の issue をすべて見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。