[BUG] Barrier (QP) is not run-to-run deterministic on 26.08 even with cudss_deterministic=1; 26.2.0 was bitwise deterministic

未关闭
  • 哪里不对？
#2,113 4 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
39/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
领域
backend, performance

调研方向

Start by reviewing the Barrier solver’s handling of quadratic programs and the cudss_deterministic setting, then compare the iteration behavior around iteration 8 across repeated runs. Reproduce with the reported C API setup and check both primal hashes and the external duality-gap certificate. Done means repeated solves on the same build and GPU are deterministic, or the documentation and optimality status accurately describe the observed behavior.

由索引模型根据 Issue 内容生成。

描述

Describe the bug
On cuOpt 26.08 the Barrier solver is not run-to-run deterministic on a convex QP, even with CUOPT_CUDSS_DETERMINISTIC=1. The same model, process and GPU give a different primal point on every run. Sometimes the point reported as Optimal fails an external optimality (duality-gap) check. cuOpt 26.2.0 was bitwise deterministic on the same model.

Problem

  • Convex QP with a diagonal (separable) Hessian and ranged rows, solved through the C API (cuOptCreateRangedProblem with a quadratic objective, then cuOptSolve).
  • 601,162 rows × 609,566 columns, 3.18 M constraint nonzeros, 609,566 Hessian nonzeros (diagonal), all bounds finite or IEEE infinity.
  • Statistical disaggregation model, minimum-χ² objective. The objective is strictly convex, so the optimum is unique.

Settings (set explicitly on every solve)
method=3 (Barrier), crossover=0, cudss_deterministic=1, presolve=2 (turned off by cuOpt for a QP), absolute/relative primal/dual/gap tolerances 1e-4, time_limit=3600.

Observed (15 runs, same binary and input, 26.08)

  • Objectives spread over 78457.92 – 78459.54, and every run has a different primal (FNV hash of the primal bytes).
  • The iteration logs are identical up to iteration 7 and diverge from iteration 8.
  • 2 of 15 runs returned status Optimal with a point that failed our external certificate. That certificate is a Lagrangian dual bound computed from the returned row duals. Our post-processed point then had relative gaps of 0.43 and 0.97.
  • With 26.2.0 (cuDSS 0.7.1), 3 runs gave the same primal bit for bit (identical hash; element-wise comparison: 0 of 296,448 values differ). That model was an earlier formulation of the same problem.

Settings tried on 26.08, none restores determinism

  • random_seed (fixed)
  • folding=0
  • dualize=0
  • barrier_iterative_refinement=1
  • OMP_NUM_THREADS=1
  • all six tolerances at 1e-6
  • determinism_mode: rejected for a QP (MIP only)
  • augmented=1: stalls, no convergence in more than 10 min

Expected behavior
With cudss_deterministic=1, the same model on the same GPU and build should return the same point, as it did in 26.2.0. Failing that, the documentation should say that Barrier is not deterministic outside cuDSS. Separately, a point that is not optimal to the requested tolerances should not be reported as Optimal. See also #2090, a different 26.08 regression where Optimal is reported on a wrong point.

Environment

  • cuOpt 26.08.0 (libcuopt-cu13==26.8.0), CUDA 13.4.2, driver 615.71.09
  • NVIDIA RTX 4070 Ti SUPER (sm_89, 16 GB), Linux x86_64
  • Previously deterministic: cuOpt 26.2.0 (CUDA 12 build)

Workaround
We moved the solve to a CPU interior point (HiGHS 1.15.1 HiPO, fixed thread count). It returns the same bits on every run. cuOpt is kept only as an opt-in path that falls back to the CPU solver when the certificate fails.

We can share the model privately as MPS if that helps. It is derived from public aggregate statistics.

主要语言
Cuda
星标
1.1k
派生
249
平均合并
1 天 15 小时
30 天内合并 PR
126

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/cuopt 的其他 Issue

查看 NVIDIA/cuopt 的全部 Issue

相似的 Issue

更多 Backend & API Design Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。