Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

CI: ASan/Valgrind shards cancelled mid-run — runner receives shutdown signal (likely OOM on 16GB hosted runner)

Abierto
  • ¿Algo no está bien?
#9,279 3 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Ya se ha fusionado un pull request relacionado.

  • #9280 de @nGoline — fusionado

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
c, github-actions
Área
ci-cd, performance, testing-qa

Línea de trabajo

Comienza en .github/workflows/ci.yaml, en los jobs de sanitizer y Valgrind alrededor de las líneas 665, 729, 771 y 773; inspecciona su configuración de matrix, el número de workers y el comando de prueba. Reproduce los shards afectados y, después, verifica que el ajuste de recursos elegido evita las cancelaciones por el apagado del runner mientras los shards hermanos siguen ejecutándose con normalidad.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The heaviest CI jobs (ASan/UBSan and Valgrind Test CLN) intermittently get their runner killed part-way through the test step. The GitHub annotation only shows Error: The operation was canceled, with no test failure, which makes it look like "almost every test broke". On the ASan/UBSan (3/6) shard it reproduces at almost exactly the 10% mark.

This is not a test failure and not the concurrency/fail-fast config. The runner VM itself is being terminated mid-run, most likely from memory exhaustion on the 16GB hosted runner.

Evidence

The real cause is one line above the cancellation in the raw job log (hidden from the annotations view):

##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[gw2] [ 10%] PASSED tests/test_closing.py::test_segwit_shutdown_script
##[error]The operation was canceled.

No test errored. The runner agent died, so every in-flight and pending test on that shard is reported cancelled.

Sample runs:

What it is not

  • Not the concurrency / cancel-in-progress group (.github/workflows/ci.yaml:9). That cancels the whole run. In the master run above only three shards died while every sibling passed:

    ASan/UBSan (1/6) success   ASan/UBSan (3/6) FAILURE   ASan/UBSan (4/6) success ...
Valgrind (10/12) FAILURE   Valgrind (12/12) FAILURE   (other 10 succeeded)

  • Not fail-fast. Both integration-sanitizers and integration-valgrind set fail-fast: false (ci.yaml:729, ci.yaml:665), so a sibling cannot cancel them.

  • Not a timeout. The shard died at ~6 min; the job timeout is 120 min and the per-test timeout is 1800s.

Individual runner VMs are dying independently, concentrated on exactly the two most memory-heavy job types.

Likely root cause: memory exhaustion on the 16GB hosted runner

  • runs-on: ubuntu-24.04 is a standard GitHub-hosted runner: 4 vCPU / 16GB RAM.
  • The test step runs pytest -n $(($(nproc) + 1)) -> 5 workers (ci.yaml:773; log confirms created: 5/5 workers).
  • The build under test is compile-clang-sanitizers; ASan roughly triples RSS and Valgrind is heavier still.
  • At the moment of death, gw0 had been running tests/test_askrene.py::test_real_data (loads the real mainnet gossip map, one of the most memory-hungry tests) for ~3 min alongside four other worker node-clusters.

Five parallel ASan lightningd+bitcoind clusters with test_real_data in the mix peaks past 16GB, and the host force-terminates the VM.

Why always ~10% on ASan/UBSan (3/6): --test-group-random-seed=42 (ci.yaml:771) fixes the shard's test set and order deterministically, so the shard hits its memory peak at the same point every run. Other runs hit the same wall on different heavy shards (e.g. Valgrind 10/12, 12/12).

The one thing not provable from the outside is OOM vs. a random hosted-runner reclaim, because GitHub does not expose the VM dmesg. The concentration on only ASan/Valgrind plus the deterministic repro point argues strongly for resource exhaustion rather than random reclaim.

Suggested fixes (cheapest first)

  1. Drop the worker count on the sanitizer/valgrind test steps: change -n $(($(nproc) + 1)) to -n $(nproc) or -n $(($(nproc) - 1)) for those two jobs only. Fewer concurrent node-clusters means lower peak memory.
  2. Cap ASan memory via ASAN_OPTIONS=quarantine_size_mb=64:malloc_context_size=5 in the sanitizer job env.
  3. Add an if: failure() step running dmesg | grep -i oom || journalctl -k | tail so the next failure records the actual cause.

Likely the same underlying overload as the timing-based flakes such as #9268 (an 11s delay to activate connectd under load). Related tracking issue: #9222.

Lenguaje dominante
C
Estrellas
3.1k
Forks
1k
Merge medio
4 d 2 h
PR fusionados (30 d)
45

Preparar el entorno

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ElementsProject/lightning

Todos los issues de ElementsProject/lightning

Issues similares

Más issues de C

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.