Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

retry_backoff retries the entire multi-family GPU test run even when a single family's genuine (non-transient) failure is the cause

Aperta
  • Qualcosa non va?
#1,576 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@lukiod ci sta già lavorando.

Dal 2/10/2026.

  • #1577 di @lukiod — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github-actions, python
Ambito
ci-cd

Direzione di ricerca

Start with .github/workflows/community-ci.yml, especially run_on_instance() and retry_backoff(), then inspect tools/community_gpu_ci.py and tools/brev_exec.py to understand their failure reporting. Done means setup or network failures still retry, while a clean family failure is identified and skips the full-run retry without changing the retry logic covered by issues #1562, #1569, #1571, and #1573.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Background

.github/workflows/community-ci.yml's run_on_instance() wraps the
entire python3 -m tools.brev_exec call (which runs every selected
family's container/tests, e.g. all ~130 families for scope=all) in
retry_backoff() — 6 attempts, capped exponential backoff up to 160s.
This was designed to survive transient network blips reaching
github.com/nvcr.io mid-run.

Observed in Dev Community GPU CI run 36962671513 (PR #1575): all
families that ran passed (e.g. whisper), but the newly-added family
ltx2 genuinely failed:

ERROR: Community GPU family failures: ltx2: checkpoint staging exited 1

Since tools/community_gpu_ci.py exits non-zero when any family
fails, retry_backoff could not distinguish this deterministic,
non-transient content bug from a network blip, and retried the entire
multi-family run 6 times (~8 minutes of wasted retry delay, plus
however long each full re-run takes) before finally giving up and
recreating the instance — none of which could ever fix a genuine
checkpoint-staging bug in one family's code.

Problem

retry_backoff cannot currently distinguish:

  • A step that failed before any family ran (setup/network issue —
    worth retrying)
  • A specific family's test genuinely, deterministically failing
    (content bug — retrying is pure waste, and delays the useful signal
    of "this PR's code is broken")

Suggested direction (not yet implemented)

  • Have tools/community_gpu_ci.py or tools/brev_exec.py distinguish
    these failure classes in its exit code or output (e.g. a specific
    exit code for "N family failures, 0 setup/network errors"), so the
    workflow can skip retrying when the failure is already known to be a
    genuine family failure.
  • Alternatively, inspect the result file for a "family failed cleanly"
    marker before deciding to retry.

Not in scope here

  • This is a workflow-logic question, not a fix for ltx2's actual
    checkpoint staging bug (that's PR #1575's own problem to fix).
  • Does not affect the other confirmed-working retry logic (#1562,
    #1569, #1571, #1573).
Lingua principale
Python
Stelle
276
Fork
73
Merge medio
1g 9h
PR unite (30g)
156

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/TensorRT-Model-Connect

Tutte le issue di NVIDIA/TensorRT-Model-Connect

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.