[Bug]: gpu-operator images v26.7.0+, contain an uncompressed layer; v26.7.1+ also switched to OCI manifest types breaks Quay/oc-mirror mirroring

Aperta
  • Qualcosa non va?
#3,166 2 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@rahulait ci sta già lavorando.

Dal 3/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

bug

Evidence

1. Manifest-list media type per version
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag | jq -r .mediaType
  done
=== v25.10.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.3.3 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.1 ===
application/vnd.oci.image.index.v1+json
=== v26.7.2 ===
application/vnd.oci.image.index.v1+json
2. Layer media types per version (amd64)
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag \
      | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$digest \
      | jq -r '.layers[].mediaType' | sort | uniq -c
  done
=== v25.10.0 ===
     36 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.3.3 ===
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.0 ===
      1 application/vnd.docker.image.rootfs.diff.tar
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.1 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip
=== v26.7.2 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip
3. The uncompressed layer is confirmed at the byte level, not just a label
$ TOKEN=$(curl -s "https://nvcr.io/token?service=nvcr.io&scope=repository:nvidia/gpu-operator:pull" \
    | jq -r .token)

$ amd64_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:v26.7.2 \
    | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')

$ layer_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$amd64_digest \
    | jq -r '.layers[] | select(.mediaType | contains("gzip") | not) | .digest')

$ curl -sL -H "Authorization: Bearer $TOKEN" \
    "https://nvcr.io/v2/nvidia/gpu-operator/blobs/$layer_digest" -o layer.bin
$ file -b layer.bin
POSIX tar archive (GNU)
$ tar -tvf layer.bin
etc/
etc/ssl/
etc/ssl/fipsmodule.cnf
etc/ssl/openssl-fips.cnf
etc/ssl/openssl-nofips.cnf

This same layer (containing OpenSSL/FIPS config files) is present,
uncompressed, in v26.7.0, v26.7.1, and v26.7.2. It does not exist at all
in v25.10.0 or v26.3.3.

Impact

Customers mirroring certified operator bundles (e.g. via oc-mirror) to
a disconnected Red Hat Quay registry fail with:

    [ERROR] : [Worker] error mirroring image
    docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
    error: Manifest list must be converted to type
    "application/vnd.docker.distribution.manifest.list.v2+json" ...
    "Instructed to preserve digests"

The only workaround today is a manual skopeo copy --format v2s2 --multi-arch=all re-push per image, which doesn't scale for
disconnected/air-gapped operator mirroring workflows.

Fix

  1. Identify the build step (introduced between v26.3.3 and v26.7.0) that
    adds the FIPS/OpenSSL config layer, and ensure it's gzip-compressed
    like every other layer.
  2. Revert to publishing Docker v2 Schema2 manifest/manifest-list media
    types (as in v25.10.0/v26.3.3/v26.7.0) instead of OCI-only, given the
    broader ecosystem's Docker v2s2 compatibility expectations for
    certified/catalog content.

Environment

  • Images affected: nvcr.io/nvidia/gpu-operator (and likely other
    nvidia/cloud-native/* child images referenced by the certified
    gpu-operator-bundle)
  • Versions: v25.10.0, v26.3.3, v26.7.0, v26.7.1, v26.7.2 (current latest)
  • Tools used to reproduce: skopeo 1.22.3, jq, file, tar, curl
Lingua principale
Go
Stelle
3k
Fork
592
Merge medio
3g 19h
PR unite (30g)
63

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/gpu-operator

Tutte le issue di NVIDIA/gpu-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.