CI: a container job that starts seconds after the dev image is pushed cannot pull it

Aperta
#2,385 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
docker, github-actions
Ambito
ci-cd

Direzione di ricerca

Start with ci.yml and the build dev/ci image job, then read tools/ci_tier_check.sh and its self-test. Compare the proposed pull check, attestation handling, and longer retry against the failure logs; the issue requires recording one choice and its reason. Done when the chosen check is logged on pull request, merge group, and dispatched runs, the tier check passes, and the requested Actions API count is added here.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:ci found-by:ci kind:flake

Seen on a dispatched full lane, run 44020220329 (ci.yml, lanes=all, on a branch at 5092ba79), 2026-11-17.

What happened

The job build dev/ci image pushed ghcr.io/inauguralsystems/eigenscript-dev:ci-5092ba79… and ended at 04:55:29Z. Four container jobs that need it started between 04:55:31Z and 04:55:40Z, and each failed in its "Initialize containers" step before any step of ours ran:

docker pull ghcr.io/inauguralsystems/eigenscript-dev:ci-5092ba793fbc4761cd5e1959e349d79ee086f65e
Error response from daemon: failed to resolve reference "ghcr.io/…/eigenscript-dev:ci-5092ba79…"
##[warning]Docker pull failed with exit code 1, back off 9.658 seconds before retry.
failed to copy: httpReadSeeker: failed open: could not fetch content descriptor
  sha256:6a8f0165885b4a4dedec34bb6adf8d127f561ec2eb626133159ce8d02cc568e5 (application/vnd.in-toto+json) from remote: not found
##[warning]Docker pull failed with exit code 1, back off 8.734 seconds before retry.
Error response from daemon: failed to resolve reference "ghcr.io/…/eigenscript-dev:ci-5092ba79…"
##[error]Docker pull failed with exit code 1

The failed jobs: db extension (postgres service), db extension suite (postgres service), extensions / server (http+net+gfx) and ancillary checks, replay differential (same-binary tape fidelity); and extensions (http+net+gfx suite; embed/lsp/jit-smoke) after them, as their aggregate. The container jobs that started later in the same run pulled the same tag and passed. The change under test was documents only.

What it looks like

The tag resolves a few seconds after the push returns, and the blob the second attempt could not fetch is the attestation that docker/build-push-action adds to the image index. The runner tries three times in about twenty seconds and gives up. So a job that is scheduled at once after the image job can lose a race with the registry.

Done when

  • Decided and recorded here: the image job proves the tag can be pulled before it ends (a pull by the pushed digest with a bounded retry), or the image is pushed without the attestation manifest, or the container jobs get a longer retry. With the reason for the one chosen.
  • The image job's log shows the check that was chosen, on a pull_request run, a merge_group run and a dispatched run.
  • tools/ci_tier_check.sh passes with the change, and its self-test covers the new step if it adds one.
  • A count from the Actions API, in this issue: of the last 200 runs of ci.yml, how many had a container job fail in "Initialize containers" with this message. That says how often the race is lost today.

— Claude DevBox

Lingua principale
C
Stelle
3
Fork
7
Merge medio
3h 28m
PR unite (30g)
270

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di InauguralSystems/EigenScript

Tutte le issue di InauguralSystems/EigenScript

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.