[BUG] test_parser_and_batch_solver intermittently hangs inside LP BatchSolve on CUDA 13.3

Aperta
  • Qualcosa non va?
#1,781 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@kaatish ci sta già lavorando.

Dal 24/8/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug

Describe the bug

test_lp_solver.py::test_parser_and_batch_solver intermittently hangs inside LP BatchSolve and never returns, taking the whole pytest cuopt step down when its time limit fires.

Evidence

Captured with faulthandler_timeout on a real CI runner (job 97564741474, conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4):

Timeout (0:10:00)!
Thread ... (most recent call first):
  File ".../cuopt/linear_programming/solver/solver.py", line 203 in BatchSolve
  File ".../cuopt/utilities/exception_handler.py", line 24 in func
  File ".../tests/linear_programming/test_lp_solver.py", line 595 in test_parser_and_batch_solver

solver.py:203 is return solver_wrapper.BatchSolve(...), so the block is in C++, not Python.

Frequency and scope

Seen on 13.3.0 only, across both amd64 and arm64:

config observed
13.3.0, 3.13, amd64, rtxpro6000 hung 3 of 4 runs
13.3.0, 3.14, arm64, l4 hung
13.3.0, 3.14, amd64, h100 hung once, passed on retry
13.0.3, 12.x (all) not seen

It is a full stop, not slowness: a healthy run of the suite completes in ~104s, while a hung one is still at 140/142 when killed. Raising the step limit from 30m to 45m did not change the outcome.

It is not specific to any PR -- it reproduces on a branch that changes only ci/*.sh.

What is ruled out

The test sets CUOPT_METHOD = PDLP, and solve_lp_with_method only dispatches to run_concurrent when the method is Concurrent. So the nested omp taskgroup / omp task / nested omp parallel constructs in solve.cu:1595-1690 are not on this path -- they all live inside run_concurrent. run_pdlp contains no OpenMP.

What is not yet established

The batch driver parallelises across models with #pragma omp parallel for num_threads(max_thread) (cpp/src/pdlp/utilities/cython_solve.cu:289, max_thread capped at 4), so several run_pdlp calls execute concurrently on one GPU. The blocking frame within that has not been identified -- the faulthandler stack proves Python is blocked in the Cython call but not where C++ stops.

One asymmetry worth checking when someone picks this up, flagged but unverified (cpp/src/pdlp/pdhg.cu:101-102):

graph_all{stream_view_, is_legacy_batch_mode || batch_mode_},
graph_prim_proj_gradient_dual{stream_view_, is_legacy_batch_mode},   // no batch_mode_

graph_all bypasses CUDA graphs in batch mode; graph_prim_proj_gradient_dual still captures. Capture uses cudaStreamCaptureModeThreadLocal (manual_cuda_graph.cuh:82), which is the multi-thread-safe mode, so this is a lead rather than a cause.

A native backtrace (gdb -p on a hung process) is the next step.

Note on severity

LP BatchSolve is deprecated (solver.py:193-199, "will be removed in a future release"), but it still ships, so callers can hit this today.

Test status

The test is being skipped to unblock CI. xfail does not work here: the test never returns, so pytest cannot observe a failure -- the outer timeout kills the whole step instead.

Lingua principale
Cuda
Stelle
1.1k
Fork
255
Merge medio
1g 11h
PR unite (30g)
123

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuopt

Tutte le issue di NVIDIA/cuopt

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.