conda-cpp-tests silently hangs on arm64 runners until the 60m step timeout kills it
维护者通常 1 天内回复
@ramakrishnap-nv 已经在做这个了。
开始于 2026年10月5日。
评估
这个 Issue 还没有评估数据。
描述
conda-cpp-tests has silently hung on arm64 runners on two independent occasions, each killed only by the step's 60-minute timeout (TIMEOUT: 'gtests (run_ctests.sh)' did not finish within its 60m time limit and was killed.). Neither occurrence points at a real test failure — in both cases every test that got to finish and report passed cleanly; the run just stops producing any output partway through.
Possibly related to #1976 (build-phase hangs on amd64) — same symptom class (silent runner-level stall, not a code bug), but this is test-execution phase and specifically implicates arm64, so tracking separately with cross-link.
Evidence
Three independent hangs, two different runs, two different GPU types, two different stuck tests — no shared code path, no shared trigger commit:
| Run | Commit | Arch/GPU/CUDA | Stuck test | Elapsed before kill |
|---|---|---|---|---|
| #2030 job 111931458886 | 4f75c1f0 |
arm64, L4, CUDA 13.3.0 | termination_status.gf2_presolve_infeasible |
~48 min silent, 60 min total |
| #2030 job 111931458869 | 4f75c1f0 |
arm64, L4, CUDA 13.0.3 | mip_solve.fixed_problem_infeasible_test |
~50 min silent, 60 min total |
main "test" workflow, run 37271359718, job 111638906301 |
81944425 (independent of #2030) |
arm64, L4, CUDA 12.2.2 | termination_status.gf2_presolve_infeasible — same test as the #2030 L4/13.3.0 run above |
~47 min silent, 60 min total |
main "test" workflow, run 37271359718, job 111638906319 |
81944425 |
arm64, gh200, CUDA 13.3.0 | pdlp_class.run_double_very_low_accuracy — unrelated test, unrelated code path |
~54 min silent, 60 min total |
The termination_status.gf2_presolve_infeasible match (same test, same L4 GPU type, two unrelated commits hours apart) at first looked like a single flaky test. The gh200 row rules that out: a completely different, unrelated test (pdlp_class.run_double_very_low_accuracy, numerics-only, no MIP/presolve/infeasibility code in common) hung on a different arm64 GPU type in the very same main run. Neither test has been recently touched (both files last modified months ago, well before any of these commits). Every amd64 leg in both runs passed normally in 30-45 minutes.
Conclusion: the stuck test varies by run and GPU type — the signature of a runner-level stall (resource contention, a hung subprocess, a network/disk stall), not a bug in any specific test or in the PRs whose CI happened to be running at the time. #2030 in particular is cleared: its diff is a rmm::cuda_stream_view → cuda::stream_ref type-signature swap nowhere near MIP presolve/infeasibility or PDLP numerics.
Ask
Flagging for the CI/infra team to correlate against arm64 runner-pool health/load at these timestamps (same ask as #1976 for amd64). No code-side fix is actionable from our side — these are hangs with zero diagnostic output, not assertion failures.
- 主要语言
- Cuda
- 星标
- 1.1k
- 派生
- 249
- 平均合并
- 1 天 15 小时
- 30 天内合并 PR
- 124
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。