[BUG] MILP intermittently segfaults (exit 139) on a small Mean-CVaR model under default settings, 26.08, RTX PRO 6000 Blackwell

Aperta
#2,045 9 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Kh4ster ci sta già lavorando.

Dal 5/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Describe the bug

A small Mean-CVaR MILP (10,063 variables, 31 binary; 10,033 constraints, 330,124 nonzeros) intermittently crashes the process with SIGSEGV (exit 139) under default settings. Runs that do finish always return Optimal with the same objective, so the model is fine: the crash is no answer, never a wrong one.

Observed on one workstation, same script, same seed:

Setting Segfaults Solve time when it finishes
defaults (standalone script below) 3 of 12 16.8 to 23.1 s
defaults (same model inside a larger harness) 5 of 12 16.0 to 23.9 s
num_cpu_threads=8 1 of 8 23.4 to 33.1 s
mip_determinism_mode=1 0 of 8 126.0 to 142.7 s

python -X faulthandler prints about 33 simultaneous Fatal Python error: Segmentation fault headers and no Python frame, which points at native solver threads. HiGHS solves the identical model run on its own with 0 crashes in 3 runs (about 57 s each), reaching the same objective.

Possibly related: #438 (also signal 11 in MIP), though that report hits an LP numerical issue first and is on an older release.

Steps/Code to reproduce bug

Synthetic data, fixed seed. Run repeatedly:

for i in $(seq 1 12); do python repro.py; echo "exit=$?"; done
#!/usr/bin/env python3
"""Minimal reproducer: cuOpt MILP intermittently segfaults on a small Mean-CVaR problem.

Synthetic data, fixed seed, defaults except time limit and MIP gap. Run it repeatedly:
    for i in $(seq 1 12); do python repro_cuopt_milp_segfault.py; echo "exit=$?"; done
Observed: exit 139 (SIGSEGV) in a fraction of runs; runs that finish report Optimal with
the same objective every time.
"""
import numpy as np
from scipy import sparse
from cuopt import linear_programming as lp
from cuopt.linear_programming.solver.solver_parameters import (
    CUOPT_LOG_TO_CONSOLE, CUOPT_MIP_RELATIVE_GAP, CUOPT_TIME_LIMIT)

N, S, K, ALPHA, CAP = 31, 10_000, 5, 0.95, 0.35
rng = np.random.default_rng(20260912)
mu = rng.uniform(0.02, 0.11, N)
vol = rng.uniform(0.04, 0.24, N)
load = rng.normal(0.0, 0.6, (N, 4))
raw = (rng.standard_t(5, (S, 4)) / np.sqrt(5 / 3)) @ load.T + rng.standard_t(5, (S, N)) / np.sqrt(5 / 3)
rets = mu + raw / raw.std(axis=0) * vol

# Variables [w (N), zeta, u (S), z (N binary)]: min -mu'w + zeta + sum(u)/((1-a)S)
c = np.concatenate([-mu, [1.0], np.full(S, 1 / ((1 - ALPHA) * S)), np.zeros(N)])
rows = sparse.vstack([
    sparse.hstack([-rets, -np.ones((S, 1)), -sparse.eye(S), sparse.csr_matrix((S, N))]),  # u_s >= -r_s'w - zeta
    sparse.hstack([np.ones((1, N)), sparse.csr_matrix((1, 1 + S + N))]),                 # sum w = 1
    sparse.hstack([sparse.eye(N), sparse.csr_matrix((N, 1 + S)), -CAP * sparse.eye(N)]),  # w_i <= cap z_i
    sparse.hstack([sparse.csr_matrix((1, N + 1 + S)), np.ones((1, N))]),                 # sum z <= K
], format="csr")
lo = np.concatenate([np.full(S, -np.inf), [1.0], np.full(N, -np.inf), [-np.inf]])
hi = np.concatenate([np.zeros(S), [1.0], np.zeros(N), [float(K)]])

dm = lp.DataModel()
dm.set_csr_constraint_matrix(rows.data, rows.indices.astype(np.int32), rows.indptr.astype(np.int32))
dm.set_constraint_lower_bounds(lo)
dm.set_constraint_upper_bounds(hi)
dm.set_objective_coefficients(c)
dm.set_variable_lower_bounds(np.concatenate([np.zeros(N), [-np.inf], np.zeros(S), np.zeros(N)]))
dm.set_variable_upper_bounds(np.concatenate([np.full(N, CAP), [np.inf], np.full(S, np.inf), np.ones(N)]))
dm.set_variable_types(np.array(["C"] * (N + 1 + S) + ["I"] * N))
ss = lp.SolverSettings()
ss.set_parameter(CUOPT_TIME_LIMIT, 300)
ss.set_parameter(CUOPT_MIP_RELATIVE_GAP, 1e-4)
ss.set_parameter(CUOPT_LOG_TO_CONSOLE, False)
sol = lp.Solve(dm, ss)
print(sol.get_termination_reason(), sol.get_primal_objective(), round(sol.get_solve_time(), 2))

Expected behavior

Either Optimal every run, or an error status. Never a process crash.

Environment details

  • cuOpt: cuopt-cu13 26.08.0 (libcuopt-cu13 26.8.0, rmm-cu13 26.8.0), pip wheels
  • Python 3.12, numpy 2.4.6, scipy 1.18.1
  • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (compute capability 12.0), driver 580.126.09, CUDA 13 userspace
  • CPU: AMD Threadripper PRO 9985WX, 64 cores / 128 threads
  • OS: openSUSE Leap 16.0, kernel 6.12
  • Other processes held GPU memory on the same card; about 9.7 GB was free when these runs started. Device memory (default RMM resource), not managed memory.

Additional context

Solving through cuopt.linear_programming.DataModel + Solve, time_limit=300, mip_relative_gap=1e-4, console log off. A separate run of the same model returned NoTermination with no primal vector and no error message once in 8 runs; that may or may not be the same defect.

Lingua principale
Cuda
Stelle
1.1k
Fork
249
Merge medio
1g 15h
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuopt

Tutte le issue di NVIDIA/cuopt

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.