[Bug]: gpu-operator images v26.7.0+, contain an uncompressed layer; v26.7.1+ also switched to OCI manifest types breaks Quay/oc-mirror mirroring

未关闭
  • 哪里不对？
#3,166 2 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@rahulait 已经在做这个了。

开始于 2026年11月3日。

评估

这个 Issue 还没有评估数据。

描述

bug

Evidence

1. Manifest-list media type per version
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag | jq -r .mediaType
  done
=== v25.10.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.3.3 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.0 ===
application/vnd.docker.distribution.manifest.list.v2+json
=== v26.7.1 ===
application/vnd.oci.image.index.v1+json
=== v26.7.2 ===
application/vnd.oci.image.index.v1+json
2. Layer media types per version (amd64)
$ for tag in v25.10.0 v26.3.3 v26.7.0 v26.7.1 v26.7.2; do
    echo "=== $tag ==="
    digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:$tag \
      | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')
    skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$digest \
      | jq -r '.layers[].mediaType' | sort | uniq -c
  done
=== v25.10.0 ===
     36 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.3.3 ===
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.0 ===
      1 application/vnd.docker.image.rootfs.diff.tar
     42 application/vnd.docker.image.rootfs.diff.tar.gzip
=== v26.7.1 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip
=== v26.7.2 ===
      1 application/vnd.oci.image.layer.v1.tar
     42 application/vnd.oci.image.layer.v1.tar+gzip
3. The uncompressed layer is confirmed at the byte level, not just a label
$ TOKEN=$(curl -s "https://nvcr.io/token?service=nvcr.io&scope=repository:nvidia/gpu-operator:pull" \
    | jq -r .token)

$ amd64_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator:v26.7.2 \
    | jq -r '.manifests[] | select(.platform.architecture=="amd64") | .digest')

$ layer_digest=$(skopeo inspect --raw docker://nvcr.io/nvidia/gpu-operator@$amd64_digest \
    | jq -r '.layers[] | select(.mediaType | contains("gzip") | not) | .digest')

$ curl -sL -H "Authorization: Bearer $TOKEN" \
    "https://nvcr.io/v2/nvidia/gpu-operator/blobs/$layer_digest" -o layer.bin
$ file -b layer.bin
POSIX tar archive (GNU)
$ tar -tvf layer.bin
etc/
etc/ssl/
etc/ssl/fipsmodule.cnf
etc/ssl/openssl-fips.cnf
etc/ssl/openssl-nofips.cnf

This same layer (containing OpenSSL/FIPS config files) is present,
uncompressed, in v26.7.0, v26.7.1, and v26.7.2. It does not exist at all
in v25.10.0 or v26.3.3.

Impact

Customers mirroring certified operator bundles (e.g. via oc-mirror) to
a disconnected Red Hat Quay registry fail with:

    [ERROR] : [Worker] error mirroring image
    docker://nvcr.io/nvidia/cloud-native/<image>@sha256:...
    error: Manifest list must be converted to type
    "application/vnd.docker.distribution.manifest.list.v2+json" ...
    "Instructed to preserve digests"

The only workaround today is a manual skopeo copy --format v2s2 --multi-arch=all re-push per image, which doesn't scale for
disconnected/air-gapped operator mirroring workflows.

Fix

  1. Identify the build step (introduced between v26.3.3 and v26.7.0) that
    adds the FIPS/OpenSSL config layer, and ensure it's gzip-compressed
    like every other layer.
  2. Revert to publishing Docker v2 Schema2 manifest/manifest-list media
    types (as in v25.10.0/v26.3.3/v26.7.0) instead of OCI-only, given the
    broader ecosystem's Docker v2s2 compatibility expectations for
    certified/catalog content.

Environment

  • Images affected: nvcr.io/nvidia/gpu-operator (and likely other
    nvidia/cloud-native/* child images referenced by the certified
    gpu-operator-bundle)
  • Versions: v25.10.0, v26.3.3, v26.7.0, v26.7.1, v26.7.2 (current latest)
  • Tools used to reproduce: skopeo 1.22.3, jq, file, tar, curl
主要语言
Go
星标
3k
派生
592
平均合并
3 天 19 小时
30 天内合并 PR
63

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

NVIDIA/gpu-operator 的其他 Issue

查看 NVIDIA/gpu-operator 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。