proxy network: injected CA is wiped when a build step upgrades the ca-certificates package

Open
  • Is something wrong?
#7,185 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
arch-linux, build-system, debian, docker, dockerfile, fedora, go, linux, ubuntu
Domain
build-system, cloud, devops, infrastructure, security

Research direction

The bug is in the executor's proxy CA injection logic, likely in the InjectProxyCA function. Start by examining the linuxSystemCertFiles list and the bundle injection code. The fix involves writing the CA to trust anchor directories (e.g., /usr/local/share/ca-certificates) and cleaning up by fingerprint. Look at the linked branch for the proposed implementation. Testing requires setting up a BuildKit proxy network and reproducing with a Dockerfile that upgrades ca-certificates. Verify by running curl after the upgrade in the same RUN step.

Written by the indexing model from the issue text.

Description

Well-formed report checklist:

  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem (searched proxy network CA, PIP_CERT, ca-certificates proxy)
  • I have provided version/information about my environment and done my best to provide a reproducer

Bug description

With proxy networking enabled (the feature from #6740), a build step that
upgrades the ca-certificates package silently removes the injected CA, and
every later HTTPS request in the same RUN fails.

InjectProxyCA appends the CA inline (marker-wrapped) to the first existing
bundle from linuxSystemCertFiles — always an extracted bundle:

  • Debian/Ubuntu/Alpine: /etc/ssl/certs/ca-certificates.crt (generated by
    update-ca-certificates)
  • RHEL-family/openEuler: /etc/pki/tls/certs/ca-bundle.crt (symlink into
    /etc/pki/ca-trust/extracted/, generated by update-ca-trust extract)

Upgrading the ca-certificates package re-runs the update tool, which
rebuilds the extracted bundle from the source directories. The BuildKit CA
is in none of them, so the regenerated bundle drops it:

Distro Regeneration tool Auto trigger Anchor dir
Debian/Ubuntu update-ca-certificates deb postinst /usr/local/share/ca-certificates
RHEL/openEuler update-ca-trust extract (p11-kit) rpm install /etc/pki/ca-trust/source/anchors
Alpine update-ca-certificates apk trigger /usr/local/share/ca-certificates
Arch update-ca-trust (p11-kit) pacman hook /etc/ca-certificates/trust-source/anchors
SUSE p11-kit package upgrade /etc/pki/trust/anchors

References: update-ca-certificates(8)
("generates ca-certificates.crt, a concatenated single-file list"),
update-ca-trust(8) ("this
command is being run during rpm package installation"; "add it as a new file
to directory /etc/pki/ca-trust/source/anchors"),
Alpine APKBUILD
(triggers=ca-certificates.trigger=...),
SUSE spec.

Note that the trigger is automatic: any package operation that upgrades
ca-certificates as a dependency (e.g. RUN apt-get install curl) wipes the
CA without any explicit update-ca-* call. Only commands inside the same exec
are affected; each RUN re-injects.

Observed in production CI on openEuler 24.03 (yum update -y && curl https://repo.openeuler.org/... → curl: (60) SSL certificate problem: unable to get local issuer certificate).

Reproduction

Reproducer runs in containers (BuildKit v0.33.0 has the proxy network feature,
so no external proxy setup is needed):

docker buildx create --name bk-proxy --driver docker-container --bootstrap
export BUILDX_BUILDER=bk-proxy

# Dockerfile — deb postinst triggers update-ca-certificates, wiping the CA,
# then curl in the same RUN fails
FROM debian:bookworm-slim
RUN apt-get update \
  && apt-get install -y --no-install-recommends ca-certificates curl \
  && curl -fsSL https://deb.debian.org/ -o /dev/null

#35 4.366 curl: (60) SSL certificate problem: unable to get local issuer certificate
#35 4.366 More details here: https://curl.se/docs/sslcerts.html

Expected: the CA injected at exec start stays trusted for the whole exec, so
the curl succeeds. Equivalently reproducible with update-ca-certificates --fresh (Debian) or update-ca-trust extract (RHEL-family) as the
regeneration command, and with yum/apk/pacman package upgrades on their
respective distros.

Version information

buildkitd github.com/moby/buildkit v0.33.0 (upstream release)
buildctl github.com/moby/buildkit v0.33.0
rootful OCI worker, Linux

Proposed fix

In addition to the inline bundle injection, write the CA into the trust
anchor directories that exist in the rootfs:

/usr/local/share/ca-certificates          (Debian, Ubuntu, Alpine)
/etc/pki/ca-trust/source/anchors          (RHEL, Fedora, openEuler)
/etc/pki/trust/anchors                    (SUSE, openSUSE)
/etc/ca-certificates/trust-source/anchors (Arch)

Regeneration scans these directories and picks the CA back up, so trust is
preserved for the rest of the exec. Cleanup then removes the anchor files and
strips the CA from the bundles by certificate fingerprint (marker block
first, fingerprint fallback for the regenerated case), which preserves
legitimate changes the exec made — e.g. an actual ca-certificates upgrade
must not be rolled back by restoring a pre-exec copy of the bundle.

Classic bundle paths that are symbolic links (RHEL-family, Arch, SUSE) need
no special handling: fs.RootPath already resolves them to their targets
within the rootfs, so the links themselves are never touched.

Implemented on top of v0.33.0 with unit tests and docs: branch
v0.33.0-ca-inject,
commit f63b1cac1 (executor: write proxy CA into trust anchor directories).
Happy to split it into upstreamable patches.

cc @tonistiigi (author of #6740)

Dominant language
Go
Stars
10.4k
Forks
1.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from moby/buildkit

All issues in moby/buildkit

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.