conda-cpp-tests silently hangs on arm64 runners until the 60m step timeout kills it

Aperta
  • Qualcosa non va?
#2,052 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@ramakrishnap-nv ci sta già lavorando.

Dal 5/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

awaiting response

conda-cpp-tests has silently hung on arm64 runners on two independent occasions, each killed only by the step's 60-minute timeout (TIMEOUT: 'gtests (run_ctests.sh)' did not finish within its 60m time limit and was killed.). Neither occurrence points at a real test failure — in both cases every test that got to finish and report passed cleanly; the run just stops producing any output partway through.

Possibly related to #1976 (build-phase hangs on amd64) — same symptom class (silent runner-level stall, not a code bug), but this is test-execution phase and specifically implicates arm64, so tracking separately with cross-link.

Evidence

Three independent hangs, two different runs, two different GPU types, two different stuck tests — no shared code path, no shared trigger commit:

Run Commit Arch/GPU/CUDA Stuck test Elapsed before kill
#2030 job 111931458886 4f75c1f0 arm64, L4, CUDA 13.3.0 termination_status.gf2_presolve_infeasible ~48 min silent, 60 min total
#2030 job 111931458869 4f75c1f0 arm64, L4, CUDA 13.0.3 mip_solve.fixed_problem_infeasible_test ~50 min silent, 60 min total
main "test" workflow, run 37271359718, job 111638906301 81944425 (independent of #2030) arm64, L4, CUDA 12.2.2 termination_status.gf2_presolve_infeasible — same test as the #2030 L4/13.3.0 run above ~47 min silent, 60 min total
main "test" workflow, run 37271359718, job 111638906319 81944425 arm64, gh200, CUDA 13.3.0 pdlp_class.run_double_very_low_accuracy — unrelated test, unrelated code path ~54 min silent, 60 min total

The termination_status.gf2_presolve_infeasible match (same test, same L4 GPU type, two unrelated commits hours apart) at first looked like a single flaky test. The gh200 row rules that out: a completely different, unrelated test (pdlp_class.run_double_very_low_accuracy, numerics-only, no MIP/presolve/infeasibility code in common) hung on a different arm64 GPU type in the very same main run. Neither test has been recently touched (both files last modified months ago, well before any of these commits). Every amd64 leg in both runs passed normally in 30-45 minutes.

Conclusion: the stuck test varies by run and GPU type — the signature of a runner-level stall (resource contention, a hung subprocess, a network/disk stall), not a bug in any specific test or in the PRs whose CI happened to be running at the time. #2030 in particular is cleared: its diff is a rmm::cuda_stream_view → cuda::stream_ref type-signature swap nowhere near MIP presolve/infeasibility or PDLP numerics.

Ask

Flagging for the CI/infra team to correlate against arm64 runner-pool health/load at these timestamps (same ask as #1976 for amd64). No code-side fix is actionable from our side — these are hangs with zero diagnostic output, not assertion failures.

Lingua principale
Cuda
Stelle
1.1k
Fork
249
Merge medio
1g 15h
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/cuopt

Tutte le issue di NVIDIA/cuopt

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.