conda-cpp-tests silently hangs on arm64 runners until the 60m step timeout kills it

Abierto
  • ¿Algo no está bien?
#2,052 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@ramakrishnap-nv ya está trabajando en esto.

Desde el 5/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

awaiting response

conda-cpp-tests has silently hung on arm64 runners on two independent occasions, each killed only by the step's 60-minute timeout (TIMEOUT: 'gtests (run_ctests.sh)' did not finish within its 60m time limit and was killed.). Neither occurrence points at a real test failure — in both cases every test that got to finish and report passed cleanly; the run just stops producing any output partway through.

Possibly related to #1976 (build-phase hangs on amd64) — same symptom class (silent runner-level stall, not a code bug), but this is test-execution phase and specifically implicates arm64, so tracking separately with cross-link.

Evidence

Three independent hangs, two different runs, two different GPU types, two different stuck tests — no shared code path, no shared trigger commit:

Run Commit Arch/GPU/CUDA Stuck test Elapsed before kill
#2030 job 111931458886 4f75c1f0 arm64, L4, CUDA 13.3.0 termination_status.gf2_presolve_infeasible ~48 min silent, 60 min total
#2030 job 111931458869 4f75c1f0 arm64, L4, CUDA 13.0.3 mip_solve.fixed_problem_infeasible_test ~50 min silent, 60 min total
main "test" workflow, run 37271359718, job 111638906301 81944425 (independent of #2030) arm64, L4, CUDA 12.2.2 termination_status.gf2_presolve_infeasible — same test as the #2030 L4/13.3.0 run above ~47 min silent, 60 min total
main "test" workflow, run 37271359718, job 111638906319 81944425 arm64, gh200, CUDA 13.3.0 pdlp_class.run_double_very_low_accuracy — unrelated test, unrelated code path ~54 min silent, 60 min total

The termination_status.gf2_presolve_infeasible match (same test, same L4 GPU type, two unrelated commits hours apart) at first looked like a single flaky test. The gh200 row rules that out: a completely different, unrelated test (pdlp_class.run_double_very_low_accuracy, numerics-only, no MIP/presolve/infeasibility code in common) hung on a different arm64 GPU type in the very same main run. Neither test has been recently touched (both files last modified months ago, well before any of these commits). Every amd64 leg in both runs passed normally in 30-45 minutes.

Conclusion: the stuck test varies by run and GPU type — the signature of a runner-level stall (resource contention, a hung subprocess, a network/disk stall), not a bug in any specific test or in the PRs whose CI happened to be running at the time. #2030 in particular is cleared: its diff is a rmm::cuda_stream_view → cuda::stream_ref type-signature swap nowhere near MIP presolve/infeasibility or PDLP numerics.

Ask

Flagging for the CI/infra team to correlate against arm64 runner-pool health/load at these timestamps (same ask as #1976 for amd64). No code-side fix is actionable from our side — these are hangs with zero diagnostic output, not assertion failures.

Lenguaje dominante
Cuda
Estrellas
1.1k
Forks
249
Merge medio
1 d 15 h
PR fusionados (30 d)
124

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/cuopt

Todos los issues de NVIDIA/cuopt

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.