Build context for a Dev Container Feature is invalidated on every build

Start with the feature-starter reproduction, its devcontainer.json, and the devcontainer build --docker-path=/usr/bin/podman entry point. Compare repeated builds and inspect the feature tarball directories under /tmp/devcontainercli-foo/container-features; done means an unchanged build reuses the feature installation layer while relevant input changes still invalidate it.

Current behavior

The build context for a Dev Container Feature is invalidated on every build, even when there are no changes to the Containerfile, devcontainer.json, or any files included in the build context.

As a result, the feature installation layer is rebuilt every time instead of reusing the existing cache.

Expected behavior

The Dev Container Feature layer should reuse the build cache whenever possible and only be rebuilt when relevant inputs have changed.

Environment

Steps to reproduce

cd /tmp
git clone https://github.com/devcontainers/feature-starter.git
cd feature-starter

Remove the docker-in-docker feature and add the following feature to devcontainer.json:

"ghcr.io/devcontainers/feature-starter/color:1": {
    "favorite": "green"
}

Build the dev container:

devcontainer build --docker-path=/usr/bin/podman

Build it again without making any changes:

devcontainer build --docker-path=/usr/bin/podman

Notice that the layer responsible for installing the feature is rebuilt instead of being restored from cache:

[2/2] STEP 5/9: RUN echo "_CONTAINER_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'root' || grep -E '^root|^[^:]*:[^:]*:root:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env && echo "_REMOTE_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'node' || grep -E '^node|^[^:]*:[^:]*:node:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env
--> Using cache c466b018334665827c5d83c2c00f910ce7f80e4bff6b0eb508ac254590faffba
--> c466b0183346

[2/2] STEP 6/9: RUN --mount=type=bind,from=dev_containers_feature_content_source,source=color_0,target=/tmp/build-features-src/color_0 \
    cp -ar /tmp/build-features-src/color_0 /tmp/dev-container-features \
 && chmod -R 0755 /tmp/dev-container-features/color_0 \
 && cd /tmp/dev-container-features/color_0 \
 && chmod +x ./devcontainer-features-install.sh \
 && ./devcontainer-features-install.sh \
 && rm -rf /tmp/dev-container-features/color_0

===========================================================================
Feature       : My Favorite Color
Description   : A feature to remind you of your favorite color
Id            : ghcr.io/devcontainers/feature-starter/color
Version       : 1.0.3
Options       :
    FAVORITE="green"
===========================================================================

Activating feature 'color'
The provided favorite color is: green
...
...

Additional observations

I am not sure exactly how feature tarballs are fetched and unpacked internally, but it appears that the feature tarball is downloaded on every invocation and extracted into a versioned directory under /tmp whose name changes each run.

For example:

$ ls -lh /tmp/devcontainercli-foo/container-features
total 52K
drwxrwxr-x 4 mv mv 4.0K Jul  6 19:15 0.87.0-1783358140200
drwxrwxr-x 4 mv mv 4.0K Jul  6 19:16 0.87.0-1783358186697
drwxrwxr-x 4 mv mv 4.0K Jul  6 19:17 0.87.0-1783358228464

This changing directory name is most likely to be contributing to cache invalidation.