Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

bakery ci publish: intermittent digest 'not found' from GHCR (eventual consistency, no retry in publish path)

Open
  • Is something wrong?
#591 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
65/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
python
Domain
ci-cd, cli, devops

Research direction

Start in posit_bakery/plugins/builtin/oras/oras.py at OrasCommand.run and in posit_bakery/plugins/builtin/soci/soci.py at SociCommand.run, then trace publish() in posit_bakery/cli/ci.py. Add tests that simulate a transient not-found response followed by success. Done means registry-touching calls retry transient failures with configurable backoff while permanent errors still fail fast.

Written by the indexing model from the issue text.

Description

bug cvp:0 docker tdp:3

Summary

bakery ci publish (formerly bakery ci merge) intermittently fails with a digest-addressed manifest reported as not found by GHCR. The failures are ephemeral — re-running the merge job alone almost always succeeds — which points squarely at GHCR read-after-write (eventual consistency) on manifests that we push by digest, combined with the fact that there is no retry/backoff anywhere in the publish path.

Example failures
  • SOCI convert: SOCI convert failed for 'ImageTarget(image='workbench', version='2026.05.0+218.pro1', ...)': Error from source registry for "ghcr.io/posit-dev/workbench/tmp:workbench-...ba39112170": sha256:7e8001fdfc...: not found (run 27354249233)
  • oras fetch: Error response from registry: could not find the manifest ghcr.io/posit-dev/workbench-session-init/tmp:latest@sha256:9a5a07a1...: not found (run 27024890914)
  • oras fetch: ... ghcr.io/posit-dev/workbench-session-init/tmp:latest@sha256:a734c599...: not found (run 27024930267)

Root cause analysis

The mechanism

  1. Per-platform builds push by digest (untagged). In posit_bakery/image/image_target.py (build()), when a temp registry is set, the build output is overridden to:

    output = {"type": "image", "push-by-digest": True, "name-canonical": True, "push": True}

    Each platform's manifest lands in ghcr.io/posit-dev/<image>/tmp addressable only by digest (no tag). The digest is recorded into the per-platform metadata artifact (image_metadata.py → <repo>/tmp@sha256:...).

  2. The platforms build on separate runners. Per .github/workflows/bakery-build-native.yml, build-test is a matrix with linux/amd64 and linux/arm64 on different runners (different machine class/region). Each pushes its manifest + blobs to GHCR independently.

  3. The merge job references those digests immediately. bakery ci publish (posit_bakery/cli/ci.py) then, by digest:

    • Phase 1 oras manifest index create — builds the index from get_merge_sources() (the per-platform ...@sha256: refs).
    • Phase 2 SOCI convert — oras cp <index> --to-oci-layout pulls the index and every child manifest/blob down.
    • Phase 3/4 index-copy + verify.

  4. There is zero retry/backoff anywhere in this path. Every subprocess.run in oras.py / soci.py raises hard on the first non-zero exit.

Why it's intermittent — and why a re-run fixes it

All observed errors are the same shape: a digest-addressed manifest reported not found by GHCR. The decisive clue is that re-running the merge job alone succeeds — it re-references the exact same digests. If those manifests had been garbage-collected, the re-run would fail identically. It doesn't, so the manifests are durably present; they were merely transiently unreadable when the merge runner first asked for them.

That is the signature of GHCR eventual consistency: a manifest/blob pushed by digest from runner A isn't guaranteed to be immediately readable by digest from runner B. Pushing by digest with no tag is the worst case — there's no tag write to act as a consistency checkpoint, and oras does HEAD/GET-by-digest lookups that 404 until the backend replicates. By the time the job is re-run, propagation has completed.

(This is diagnosis-by-evidence, not a reproduced failure — GHCR's internal replication can't be reproduced on demand. But the evidence chain — always digest-not-found, always self-heals on plain re-run, separate-runner pushes, no retry — is consistent and rules out permanent GC/deletion.)

Proposed solutions

Ranked. The clean fix lives in bakery, not the workflow.

  1. Retry-with-backoff around the registry-touching oras/soci calls (recommended core fix). Wrap .run() of OrasManifestIndexCreate, the SOCI oras cp --to-oci-layout pull, OrasCopy, and OrasManifestFetch so that transient registry errors (not found / manifest unknown / 5xx / timeouts) retry a handful of times with exponential backoff (e.g. 5 attempts, 2s→32s). Non-transient errors still fail fast. This directly absorbs propagation lag. Retry count/backoff should be configurable.

  2. Explicit pre-flight wait in publish (complements #1, better diagnostics). Before Phase 1, poll each source digest with oras manifest fetch --descriptor until all resolve (or timeout). Turns "hope it's propagated" into condition-based waiting and logs which digest was lagging.

  3. Workflow-level retry (stopgap only). Wrap the Publish step with nick-fields/retry. Cheap, but re-runs the whole publish and masks the real fix.

  4. Push per-platform images to real tags (tmp:<uid>-<platform>) instead of by-digest. Would also improve consistency, but a bigger change with its own GC/cleanup implications. Not the lead recommendation.

Recommendation: implement #1 as the core fix (with retry/backoff configurable), optionally #2 for diagnostics. Both should be developed with tests that simulate a transient not found followed by success.

Affected files

  • posit-bakery/posit_bakery/plugins/builtin/oras/oras.py — OrasCommand.run, index-create/copy/fetch workflows
  • posit-bakery/posit_bakery/plugins/builtin/soci/soci.py — SociCommand.run, SociConvertWorkflow
  • posit-bakery/posit_bakery/cli/ci.py — publish() orchestration (pre-flight wait would go here)
Dominant language
Python
Stars
2
Forks
0
Avg merge
2d 23h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from posit-dev/images-shared

All issues in posit-dev/images-shared

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.