Image exported from `Merge(base, Diff(base, base.Run(...)))` intermittently loses its base layers when pulled again on the same daemon (`stat /bin/sh: no such file or directory`)

Start with cache/blobs.go around computeChainMetadata and cache/manager.go around GetByBlob, then trace how cache/refs.go builds the Diff layer chain and how source/containerimage/pull.go resolves exported layers. Reproduce with the provided LLB generator and loop on v0.34.1; the issue reports that the Diff record wins when its ID sorts first. Done means the reproduction consistently pulls the complete base and patch layers, including with existing cache records.

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

After BuildKit exports an image built as Merge([base, Diff(base, base.Run(...))]), a later build FROM that same image, on the same buildkitd, sometimes gets an image that contains only the diff layer. The base layers are gone, and the first RUN fails with:

runc run failed: unable to start container process: error during container init: exec: "/bin/sh": stat /bin/sh: no such file or directory

When the exported image is pulled, its layers resolve to cache records through GetByBlob. The export leaves three cache records with the same chainID and blobChainID: the exec record, the diff record and the merge record. GetByBlob returns the first one in bolt key order. That order follows the record IDs, which are random. When the diff record sorts first, the pulled image resolves to the diff record, whose content and layer chain are only the diff.

In our runs, the outcome is fully predicted by which of the three records has the lowest ID: 67 out of 67 runs (47 with the minimal reproduction below, 20 with Copa).

This pattern is what Copa builds for RPM-based images (pkg/pkgmgr/rpm.go:661-662 at v0.15.0: patchDiff := llb.Diff(ImageState, installed), llb.Merge([]llb.State{ImageState, patchDiff})). We hit it in CI: we patch an image with Copa, then build FROM the patched image on the same buildkitd.

Expected: building FROM <exported image> always gets the full image (base layers plus the patch layer).

Observed: about 1 time in 3, the image resolves to the diff record. Re-exporting it gives a 1-layer image, and any RUN fails because /bin/sh is missing.

Reproduction (tested on v0.34.1)

1. LLB generator: Merge([base, Diff(base, base.Run(...))]), with a nonce so that every run creates new cache records. We built it against github.com/moby/buildkit v0.31.1. Any version with llb.Diff/llb.Merge should do.

   // gen/main.go — go mod init repro && go get github.com/moby/[email protected] && CGO_ENABLED=0 go build -o gen .
   package main
   
   import (
   	"context"
   	"flag"
   	"os"
   
   	"github.com/moby/buildkit/client/llb"
   )
   
   func main() {
   	image := flag.String("image", "docker.io/library/alpine:3.20", "single-layer base image")
   	nonce := flag.String("nonce", "1", "makes the RUN unique, so that every run creates new cache records")
   	flag.Parse()
   
   	base := llb.Image(*image)
   	upper := base.Run(llb.Shlexf(`sh -c 'echo %s > /nonce'`, *nonce)).Root()
   	patched := llb.Merge([]llb.State{base, llb.Diff(base, upper)})
   
   	def, err := patched.Marshal(context.Background(), llb.LinuxAmd64)
   	if err != nil {
   		panic(err)
   	}
   	if err := llb.WriteTo(def, os.Stdout); err != nil {
   		panic(err)
   	}
   }
   

2. A local registry and a fresh rootful buildkitd (OCI worker, overlayfs):

   printf '[registry."localhost:5000"]\n  http = true\n' > buildkitd.toml
   docker run -d --name repro-registry --network host -e REGISTRY_HTTP_ADDR=localhost:5000 registry:3
   docker run -d --name repro-buildkitd --network host --privileged \
     -v "$PWD/buildkitd.toml:/etc/buildkit/buildkitd.toml:ro" moby/buildkit:v0.34.1
   

3. The loop. Each run exports the merge to the registry, lists the cache records the run created in ID order, then builds FROM the pushed image:

   #!/bin/sh
   # repro.sh N — needs buildctl, BUILDKIT_HOST and ./gen
   set -u
   REF=localhost:5000/repro:latest
   mkdir -p /tmp/df && printf 'FROM %s\nRUN true\n' "$REF" > /tmp/df/Dockerfile
   for i in $(seq 1 "${1:-20}"); do
     buildctl du -v | awk '/^ID:/{print $2}' | sort > /tmp/before
     ./gen -nonce "$(date +%s%N)-$i" | buildctl build --output type=image,name=$REF,push=true,registry.insecure=true > /tmp/export-$i.log 2>&1 || { echo "run $i: export failed"; continue; }
     buildctl du -v > /tmp/du-$i.txt
     awk '/^ID:/{print $2}' /tmp/du-$i.txt | sort > /tmp/after
     comm -13 /tmp/before /tmp/after > /tmp/new
     # new records of this run, in bolt key order (= LC_ALL=C order of the IDs)
     order=$(awk '/^ID:/{id=$2} /^Description:/{print id"\t"$2}' /tmp/du-$i.txt | grep -F -f /tmp/new | LC_ALL=C sort | cut -f2 | tr '\n' ',')
     if buildctl build --frontend dockerfile.v0 --local context=/tmp/df --local dockerfile=/tmp/df > /tmp/final-$i.log 2>&1; then r=OK; else r=FAIL; fi
     echo "run $i: new records (ID order)=$order build FROM pushed image=$r $(grep -m1 -o 'stat /bin/sh: no such file or directory' /tmp/final-$i.log)"
   done
   

   docker cp gen repro-buildkitd:/gen && docker cp repro.sh repro-buildkitd:/repro.sh
   docker exec -w / repro-buildkitd sh /repro.sh 20
   

   In the record list, mount is the exec record (mount / from exec sh -c echo ... > /nonce), diff the Diff record and merge the Merge record. In the first run, pulled (the base image's own record) also appears; it doesn't compete for this blob chain.

Results: three batches on fresh daemons, 15, 20 and 12 runs. In 17 of the 47 runs the diff record had the lowest ID among the three, and all 17 failed. The other 30 runs all succeeded. Excerpt:

run 1: new records (ID order)=pulled,merge,diff,mount, build FROM pushed image=OK
run 2: new records (ID order)=diff,mount,merge, build FROM pushed image=FAIL stat /bin/sh: no such file or directory
run 3: new records (ID order)=diff,merge,mount, build FROM pushed image=FAIL stat /bin/sh: no such file or directory
run 4: new records (ID order)=merge,mount,diff, build FROM pushed image=OK
run 5: new records (ID order)=mount,diff,merge, build FROM pushed image=OK

Failing build log (first batch, so the registry address differs):

#4 [1/2] FROM 127.0.0.1:5077/repro:latest@sha256:f04afd626111de4132d7079c090e0b752a167a6d55586541f515d19810d4a53d
#5 [2/2] RUN true
#5 0.049 runc run failed: unable to start container process: error during container init: exec: "/bin/sh": stat /bin/sh: no such file or directory
#5 ERROR: process "/bin/sh -c true" did not complete successfully: exit code: 1

Note on what we actually ran: the last batch (12 runs, 4 failures) used the script and commands above unchanged, except that our network has no direct Internet access. We added a docker.io mirror to buildkitd.toml and set HTTP_PROXY/HTTPS_PROXY/NO_PROXY=localhost,127.0.0.1 on the buildkitd container. The first two batches used an earlier version of the script with the same steps (another registry port, buildctl in a separate container sharing the socket).

Same result with Copa (original setup)

Copa v0.15.0 patching rockylinux:9-minimal (one RPM package per run, so that every run creates new records), then FROM <patched> with no instructions, exported, then a smoke RUN ls /, all on one warm buildkitd. 20 runs:

• 8 runs where the diff (...) record had the lowest ID: final image with 1 layer, smoke test failed with stat /bin/sh
• 12 runs where the exec or merge record had the lowest ID: final image with 2 layers, smoke test OK

Again 20 out of 20 predicted. With buildctl prune --all between the Copa export and the next build: 11 out of 11 OK, including 3 where the diff record sorted first.

Analysis

All references are to v0.34.1 (f5b59a1f).

1. The Diff reuses the upper's blob. In Diff(base, base.Run(...)), the upper is a single layer on top of the lower, so layerWalk visits the upper (the exec record) instead of the diff (cache/refs.go:556-570, case at :564). On export, computeBlobChain for the Merge recurses into the Diff, which "is just re-using the upper blob" (cache/blobs.go:86-92), and calls computeChainMetadata on every record (:314).

2. Three records end up with the same chain IDs (computeChainMetadata, cache/blobs.go:380-481):

   • exec record E (Layer, :411-433): chainID = ChainID(base, E) and blobChainID = ChainID(base, (blobE, diffIDE))
   • diff record D (Diff, :459-473), "re-using upper blob" branch (:468-471): chainID = upper.getChainID() and blobChainID = upper.getBlobChainID(), so E's values
   • merge record M (Merge, :435-458): base chain plus the layers of D.layerChain(), which is [E], so E's values again

   E and M both stand for base + E. D's content and layer chain are only E's layer, but D carries E's full chain IDs.

3. GetByBlob returns the first match. Pulling the exported image calls GetByBlob(layer, parent) for each layer (source/containerimage/pull.go:238). For the top layer, cache/manager.go:229 searches the blob-chain index. The bolt cursor walks keys <index>::<recordID> in byte order (cache/metadata/metadata.go:104-144, indexKey at :494-496), so candidates come in record-ID order, and record IDs are random. The loop at cache/manager.go:234-266 returns the first candidate whose chainID matches (:255). It doesn't check the record kind or its layer chain. All three records pass. If D comes first, the pulled image is backed by D: its rootfs is only the diff, and exporting it gives a single layer.

This fits all the runs above: failure if and only if D has the lowest ID, and prune --all (which removes the records) makes it go away.

Copa's apk/dpkg paths diff updated against installed and merge the result onto the original image. There, M's chain (base, installed) differs from D's (base, updated, installed), so by our reading they don't alias.

On master: we read master at 570aa61d7a (2026-11-10). cache/blobs.go is identical to v0.34.1, and the GetByBlob blob-chain loop is unchanged. layerWalk now calls diffReusesUpper() (cache/refs.go:575-591) with the same conditions. Since 559537d3f7 (#7364), layerDigestChain of such a Diff returns only the upper's blob, which matches D really being one layer, but D's chain IDs are still the upper's. So the bug looks present on master. We have not run master.

Suggested fix (untested, for discussion)

1. Give a blob-reusing Diff the chain IDs of a one-layer chain (cache/blobs.go:468-471). D's content is one layer, the upper's, applied on scratch:

   } else {
   	// re-using upper blob: this diff is a single layer, the upper's
   	upper := sr.diffParents.upper
   	chainID = upper.layerIdentity()
   	if chainID == "" {
   		return nil
   	}
   	blobChainID = imagespecidentity.ChainID([]digest.Digest{upper.getBlob(), upper.getDiffID()})
   }
   

   M still gets the base + E chain, because the Merge case uses the layers' identities, not D's chain ID. D then only matches a one-layer image made of that blob, which has the same content.

2. Defence in depth in GetByBlob (cache/manager.go:234-266): next to the chainID check, skip a candidate whose layer chain (layerChain(), which is what an export would produce) is not the parent's chain plus this layer. In particular, skip a Diff that reuses its upper's blob when a parent is given. This also covers records already in existing caches that carry the old chain IDs. Without it, users would have to prune after upgrading.

Workaround: run buildctl prune --all between the export and any build that uses the exported image on the same daemon.

Related

• #5954 has the same error (stat /bin/sh: no such file or directory, intermittent, cause unknown). It uses a plain Dockerfile without Diff/Merge on Docker's embedded BuildKit, so it may be a different bug. Mentioned for cross-reference only.

Version information

buildkitd github.com/moby/buildkit v0.34.1 f5b59a1f2fb25a0cb7072c398cab9aed7835da4e
buildctl github.com/moby/buildkit v0.34.1 f5b59a1f2fb25a0cb7072c398cab9aed7835da4e

• buildkitd: moby/buildkit:v0.34.1 image, rootful, --privileged, OCI worker (runc), overlayfs snapshotter, default GC settings
• Host: Linux amd64. First seen on GitLab CI runners (Kubernetes executor, buildkitd as a service container in the job pod). The runs above were on a workstation (Docker on WSL2, kernel 6.18).
• Generator: github.com/moby/buildkit v0.31.1 (client/llb), go1.25