Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Retry transient image pulls in cosign, crane, kyverno and trivy — one GHCR connection reset turns a verdict job red

Open
  • Is something wrong?
#386 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, go
Domain
ci-cd, devops, infrastructure

Research direction

Start with the four cosign, crane, kyverno and trivy modules, then read the existing retry precedent in crossplane/container.go. Trace where each tool container is created, synced and executed, and inspect how +cache="never" and CACHE_BUSTER are handled. Done means the retry scope, transport-error policy, visibility, backoff and read-only exec behavior are specified and covered without retrying verdict failures.

Written by the indexing model from the issue text.

Description

enhancement

The four supply-chain modules pull a tool image on every call, and the whole call fails on the first transport error. In stuttgart-things/schmetterpause#255 the policy-test job went red before kyverno test had even started:

failed to resolve image "ghcr.io/kyverno/kyverno-cli:v1.19.1" (platform: "linux/amd64"): failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://pkg-containers.githubusercontent.com/ghcrblobs07/blobs/sha256:bbe3980b…": read tcp 172.17.0.2:46078->185.199.110.154:443: read: connection reset by peer

Nothing else was wrong: the fixture images still resolved, the policy had not changed, and re-running only that job passed. One TCP connection failed.

In a pull request that costs a click. In a release run it means someone re-running a job by hand. Every consumer of these modules goes red for a reason unrelated to what the job checks, and each job moved onto #376–#379 adds another place where that can happen.

What is pulled today

Module Pulled on every call
cosign gcr.io/projectsigstore/cosign:v<version>, cgr.dev/chainguard/wolfi-base:latest
crane gcr.io/go-containerregistry/crane:v<version>, cgr.dev/chainguard/wolfi-base:latest
kyverno ghcr.io/kyverno/kyverno-cli:v<version>, cgr.dev/chainguard/wolfi-base:latest
trivy aquasec/trivy:<version> (Docker Hub)

None of the four retries anything. The only precedent in this repository is in shell: crossplane/container.go downloads with curl --retry 5 --retry-all-errors.

Two kinds of failure, and only one is easy

  1. Setting up the tool container: From(...) and copying the binary out. This has no side effects and decides nothing yet, so it is always safe to retry.
  2. The tool's own network calls: cosign talking to the registry and Rekor, trivy pulling the image it inventories, kyverno fetching signatures during test. These happen inside the exec whose result is the verdict.

Proposed shape

These points should be decided up front rather than discovered later:

  • Retry the setup, not the verdict. Build the tool container and Sync it in a bounded retry loop, then run the exec once. A failed pull then never reaches the part of the call whose result means something. It also means no retry can ever turn a refusal, a test mismatch or a MANIFEST_UNKNOWN into a pass.

  • Bounded and visible. A few attempts with backoff, and every retry is printed together with the error it retries on. A silent retry is a flake nobody can count.

  • Transport errors only. Retry on connection reset by peer, i/o timeout, TLS handshake timeout, unexpected EOF and HTTP 502/503/504. Do not retry a 401/403, a manifest that does not exist, or an exec that ran and exited non-zero.

  • Docker Hub is its own case. aquasec/trivy comes from Docker Hub, where the likely failure is a 429 rate limit rather than a reset, and retrying quickly makes a rate limit worse. Either back off long enough to matter, or say in the doc comment that it is not retried.

  • Retry inside the exec only where the operation is idempotent. Retrying verify, verify-attestation, digest or sbom after a transport error is harmless. sign and attest are different:

    • a retry after the Rekor upload adds a second transparency-log entry;
    • the identity token passed in may have expired by then.

    Verification copes with both, but this should be decided, not happen as a side effect. The simplest honest rule may be: retry the setup everywhere, and retry the exec only in the read-only functions.

  • Check how +cache="never" and CACHE_BUSTER interact with a retry. The second attempt has to re-run the pull or the exec, not hand back the first attempt's failure.

Who wants it

stuttgart-things/schmetterpause, whose sign-image, sign-kustomize, verify-artefacts, verify-image-tags and policy-test jobs are now calls into these four modules (stuttgart-things/schmetterpause#255; ADR-0021 there).

Refs #376, #377, #378, #379

Dominant language
Go
Stars
4
Forks
0
Avg merge
21h 47m
Merged PRs (30d)
26

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from stuttgart-things/dagger

All issues in stuttgart-things/dagger

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.