[BUG] Barrier (QP) is not run-to-run deterministic on 26.08 even with cudss_deterministic=1; 26.2.0 was bitwise deterministic

オープン
  • 何か問題がありますか？
#2,113 コメント 4 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
39/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
領域
backend, performance

調査の方向性

Start by reviewing the Barrier solver’s handling of quadratic programs and the cudss_deterministic setting, then compare the iteration behavior around iteration 8 across repeated runs. Reproduce with the reported C API setup and check both primal hashes and the external duality-gap certificate. Done means repeated solves on the same build and GPU are deterministic, or the documentation and optimality status accurately describe the observed behavior.

索引モデルが issue の本文から書いたものです。

説明

Describe the bug
On cuOpt 26.08 the Barrier solver is not run-to-run deterministic on a convex QP, even with CUOPT_CUDSS_DETERMINISTIC=1. The same model, process and GPU give a different primal point on every run. Sometimes the point reported as Optimal fails an external optimality (duality-gap) check. cuOpt 26.2.0 was bitwise deterministic on the same model.

Problem

  • Convex QP with a diagonal (separable) Hessian and ranged rows, solved through the C API (cuOptCreateRangedProblem with a quadratic objective, then cuOptSolve).
  • 601,162 rows × 609,566 columns, 3.18 M constraint nonzeros, 609,566 Hessian nonzeros (diagonal), all bounds finite or IEEE infinity.
  • Statistical disaggregation model, minimum-χ² objective. The objective is strictly convex, so the optimum is unique.

Settings (set explicitly on every solve)
method=3 (Barrier), crossover=0, cudss_deterministic=1, presolve=2 (turned off by cuOpt for a QP), absolute/relative primal/dual/gap tolerances 1e-4, time_limit=3600.

Observed (15 runs, same binary and input, 26.08)

  • Objectives spread over 78457.92 – 78459.54, and every run has a different primal (FNV hash of the primal bytes).
  • The iteration logs are identical up to iteration 7 and diverge from iteration 8.
  • 2 of 15 runs returned status Optimal with a point that failed our external certificate. That certificate is a Lagrangian dual bound computed from the returned row duals. Our post-processed point then had relative gaps of 0.43 and 0.97.
  • With 26.2.0 (cuDSS 0.7.1), 3 runs gave the same primal bit for bit (identical hash; element-wise comparison: 0 of 296,448 values differ). That model was an earlier formulation of the same problem.

Settings tried on 26.08, none restores determinism

  • random_seed (fixed)
  • folding=0
  • dualize=0
  • barrier_iterative_refinement=1
  • OMP_NUM_THREADS=1
  • all six tolerances at 1e-6
  • determinism_mode: rejected for a QP (MIP only)
  • augmented=1: stalls, no convergence in more than 10 min

Expected behavior
With cudss_deterministic=1, the same model on the same GPU and build should return the same point, as it did in 26.2.0. Failing that, the documentation should say that Barrier is not deterministic outside cuDSS. Separately, a point that is not optimal to the requested tolerances should not be reported as Optimal. See also #2090, a different 26.08 regression where Optimal is reported on a wrong point.

Environment

  • cuOpt 26.08.0 (libcuopt-cu13==26.8.0), CUDA 13.4.2, driver 615.71.09
  • NVIDIA RTX 4070 Ti SUPER (sm_89, 16 GB), Linux x86_64
  • Previously deterministic: cuOpt 26.2.0 (CUDA 12 build)

Workaround
We moved the solve to a CPU interior point (HiGHS 1.15.1 HiPO, fixed thread count). It returns the same bits on every run. cuOpt is kept only as an opt-in path that falls back to the CPU solver when the certificate fails.

We can share the model privately as MPS if that helps. It is derived from public aggregate statistics.

主要言語
Cuda
スター
1.1k
フォーク
249
平均マージ
1日 15時間
マージ済み PR（30日）
126

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/cuopt のほかの issue

NVIDIA/cuopt の issue をすべて見る

似ている issue

Backend & API Design の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。