[BUG] Barrier (QP) is not run-to-run deterministic on 26.08 even with cudss_deterministic=1; 26.2.0 was bitwise deterministic

Đang mở
  • Có gì không ổn?
#2,113 4 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
39/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Lĩnh vực
backend, performance

Hướng nghiên cứu

Start by reviewing the Barrier solver’s handling of quadratic programs and the cudss_deterministic setting, then compare the iteration behavior around iteration 8 across repeated runs. Reproduce with the reported C API setup and check both primal hashes and the external duality-gap certificate. Done means repeated solves on the same build and GPU are deterministic, or the documentation and optimality status accurately describe the observed behavior.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Describe the bug
On cuOpt 26.08 the Barrier solver is not run-to-run deterministic on a convex QP, even with CUOPT_CUDSS_DETERMINISTIC=1. The same model, process and GPU give a different primal point on every run. Sometimes the point reported as Optimal fails an external optimality (duality-gap) check. cuOpt 26.2.0 was bitwise deterministic on the same model.

Problem

  • Convex QP with a diagonal (separable) Hessian and ranged rows, solved through the C API (cuOptCreateRangedProblem with a quadratic objective, then cuOptSolve).
  • 601,162 rows × 609,566 columns, 3.18 M constraint nonzeros, 609,566 Hessian nonzeros (diagonal), all bounds finite or IEEE infinity.
  • Statistical disaggregation model, minimum-χ² objective. The objective is strictly convex, so the optimum is unique.

Settings (set explicitly on every solve)
method=3 (Barrier), crossover=0, cudss_deterministic=1, presolve=2 (turned off by cuOpt for a QP), absolute/relative primal/dual/gap tolerances 1e-4, time_limit=3600.

Observed (15 runs, same binary and input, 26.08)

  • Objectives spread over 78457.92 – 78459.54, and every run has a different primal (FNV hash of the primal bytes).
  • The iteration logs are identical up to iteration 7 and diverge from iteration 8.
  • 2 of 15 runs returned status Optimal with a point that failed our external certificate. That certificate is a Lagrangian dual bound computed from the returned row duals. Our post-processed point then had relative gaps of 0.43 and 0.97.
  • With 26.2.0 (cuDSS 0.7.1), 3 runs gave the same primal bit for bit (identical hash; element-wise comparison: 0 of 296,448 values differ). That model was an earlier formulation of the same problem.

Settings tried on 26.08, none restores determinism

  • random_seed (fixed)
  • folding=0
  • dualize=0
  • barrier_iterative_refinement=1
  • OMP_NUM_THREADS=1
  • all six tolerances at 1e-6
  • determinism_mode: rejected for a QP (MIP only)
  • augmented=1: stalls, no convergence in more than 10 min

Expected behavior
With cudss_deterministic=1, the same model on the same GPU and build should return the same point, as it did in 26.2.0. Failing that, the documentation should say that Barrier is not deterministic outside cuDSS. Separately, a point that is not optimal to the requested tolerances should not be reported as Optimal. See also #2090, a different 26.08 regression where Optimal is reported on a wrong point.

Environment

  • cuOpt 26.08.0 (libcuopt-cu13==26.8.0), CUDA 13.4.2, driver 615.71.09
  • NVIDIA RTX 4070 Ti SUPER (sm_89, 16 GB), Linux x86_64
  • Previously deterministic: cuOpt 26.2.0 (CUDA 12 build)

Workaround
We moved the solve to a CPU interior point (HiGHS 1.15.1 HiPO, fixed thread count). It returns the same bits on every run. cuOpt is kept only as an opt-in path that falls back to the CPU solver when the certificate fails.

We can share the model privately as MPS if that helps. It is derived from public aggregate statistics.

Ngôn ngữ chính
Cuda
Star
1.1k
Fork
255
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
123

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/cuopt

Tất cả issue của NVIDIA/cuopt

Issue tương tự

Thêm issue về Backend & API Design

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.