Template build: finalize phase silently discards build-layer writes to /etc/ssl/certs

packages/orchestrator/pkg/template/build/phases/finalize/configure.go の packCertBundleCmd から始め、次に finalize/builder.go と seed-certs.sh.tpl の fresh boot を追跡します。tests/integration/internal/tests/envd/ca_cert_build_test.go を実行し、JRE template のケースを再現します。build-layer の証明書パスが finalization 後も維持され、sandbox の apt 操作によって ca-certificates-java が半端な設定のまま残らなくなれば完了です。

Template build: finalize phase silently discards build-layer writes to /etc/ssl/certs, leaving ca-certificates-java permanently half-configured (every apt-get in sandboxes exits 100)

Summary

Any template that installs a JRE (directly, or transitively e.g. via libreoffice → default-jre) ships a snapshot where:

1. /etc/ssl/certs/java/ (owned by ca-certificates-java, created by dpkg during the build) is missing at sandbox runtime, and
2. ca-certificates-java is in half-configured (iF) state in the dpkg database,

even though every build step succeeded — inside the build layers, dpkg -l ca-certificates-java reports ii and the directory exists.

The consequence is that every subsequent apt-get install inside every sandbox created from the template exits with code 100 (dpkg fails while configuring the pending ca-certificates-java), even though the requested package actually installs. This is especially painful for agent workloads that install packages on demand and treat non-zero exit codes as failures. Java TLS is also broken ($JAVA_HOME/lib/security/cacerts is a dangling symlink to /etc/ssl/certs/java/cacerts).

Root cause

Three mechanisms interact:

(a) /etc/ssl/certs is a tmpfs bind mount inside the guest.
On every guest boot, envd.service runs ExecStartPre=/usr/local/bin/e2b-seed-certs (packages/orchestrator/pkg/template/build/core/rootfs/files/seed-certs.sh.tpl), which bind-mounts a tmpfs over /etc/ssl/certs, seeded from /usr/local/share/e2b/ssl-certs.tar (or cp -aL from the underlying dir). Therefore everything dpkg writes under /etc/ssl/certs during build steps goes into the tmpfs and never reaches the NBD-backed rootfs. When ca-certificates-java is installed in a build layer, dpkg unpacks /etc/ssl/certs/java into the tmpfs; the package configures fine and asserts ii — but the directory only exists in that boot's tmpfs.

(b) The finalize phase boots a fresh VM instead of resuming the last build layer.
packages/orchestrator/pkg/template/build/phases/finalize/builder.go uses layer.NewCreateSandbox(...) (line ~193). On this fresh boot, e2b-seed-certs re-seeds the tmpfs from the tar / underlying rootfs — neither of which contains java/ — so the directory silently vanishes between the last build step and finalization.

(c) packCertBundle re-triggers the broken state right before the snapshot.
The build's last guest step (packCertBundleCmd in packages/orchestrator/pkg/template/build/phases/finalize/configure.go) runs update-ca-certificates. This invokes the jks-keystore hook shipped by ca-certificates-java, which activates its dpkg trigger and — because it runs outside a maintainer script (DPKG_MAINTSCRIPT_PACKAGE unset) — executes dpkg --triggers-only --pending itself. The triggered postinst writes to /etc/ssl/certs/java/cacerts without creating the directory (arguably a Debian bug too, but the directory is package-shipped and always exists on normal systems), fails with java.io.FileNotFoundException, and dpkg records half-configured in /var/lib/dpkg/status on the rootfs. update-ca-certificates swallows the hook failure, so the build reports success and the broken state is snapshotted.

Since sandbox creation is a pure Firecracker snapshot resume (no systemd boot, no tmpfiles, no re-provisioning), the state can never self-heal at runtime.

Evidence: guest dpkg.log timeline from an affected build

16:18:19  trigproc libc-bin ...                          <- last build layer, all OK
16:18:20  trigproc dictionaries-common ...               <- ii asserted here, /etc/ssl/certs/java exists (tmpfs)
          [finalize fresh boot at 16:18:48 — tmpfs re-seeded, java/ gone]
16:18:57  status triggers-pending ca-certificates-java
16:18:57  trigproc ca-certificates-java
16:18:57  status half-configured ca-certificates-java    <- snapshotted

Failure inside the triggered postinst:

org.debian.security.UnableToSaveKeystoreException: There was a problem saving the new Java keystore.
Caused by: java.io.FileNotFoundException: /etc/ssl/certs/java/cacerts (No such file or directory)
dpkg: error processing package ca-certificates-java (--configure): ... exit status 1

And in every sandbox created from the template:

$ sudo apt-get install -y sl; echo $?
...
Errors were encountered while processing:
 ca-certificates-java
E: Sub-process /usr/bin/dpkg returned an error code (1)
100

Why this contradicts the documented contract

The comment on packCertBundleCmd states the tar must equal the trust store the guest would regenerate at boot, "including CAs added in user layers or start/ready, registered or not", and tests/integration/internal/tests/envd/ca_cert_build_test.go explicitly asserts that certs dropped during build steps survive into the baked bundle. The intent that build-layer trust-store changes persist is clear — the ca-certificates-java keystore (and the resulting dpkg state) is a case this contract misses.

Reproduction

template = (
    Template()
    .from_template("code-interpreter-v1")   # any Debian/Ubuntu base
    .apt_install(["default-jre-headless"])  # pulls in ca-certificates-java
    # optional: assert healthy state inside the build — it passes:
    .run_cmd("dpkg -l ca-certificates-java | grep '^ii' && test -d /etc/ssl/certs/java", user="root")
)

Then in a sandbox created from the built template:

dpkg -l ca-certificates-java        # -> iF  (half-configured)
ls /etc/ssl/certs/java              # -> No such file or directory
sudo apt-get install -y sl; echo $? # -> 100

Suggested fix

In packCertBundleCmd, before running update-ca-certificates:

mkdir -p /etc/ssl/certs/java   # or: for d in $(dpkg -L ca-certificates-java 2>/dev/null | grep '^/etc/ssl/certs'); do mkdir -p "$d"; done

and/or run dpkg --configure -a || true afterwards and fail the build (or at least warn) if any package is left in a broken state — today a failed trigger is silently baked into the snapshot.

A more general option: have the finalize boot seed the tmpfs additively (preserve entries present in the underlying dir / previous layer's tmpfs that are missing from the tar) so package-owned directories under /etc/ssl/certs survive the fresh boot. The current behavior means any build-layer write under /etc/ssl/certs other than the bundle itself is silently discarded, which is surprising given the tested contract above.

Workaround (for other template authors hitting apt exit 100)

In a build layer, move the Java keystore off the tmpfs and neutralize the trigger:

mkdir -p /etc/ssl/certs/java && dpkg --configure -a
update-ca-certificates -f || true
mkdir -p /usr/local/share/java-cacerts
cp -L /etc/ssl/certs/java/cacerts /usr/local/share/java-cacerts/cacerts
JHOME=$(dirname $(dirname $(readlink -f /usr/bin/java)))
ln -sfn /usr/local/share/java-cacerts/cacerts "$JHOME/lib/security/cacerts"
sed -i 's/^cacerts_updates=.*/cacerts_updates=no/' /etc/default/cacerts

With cacerts_updates=no, the finalize-time trigger becomes a successful no-op, the package stays ii, and Java reads its trust store from a rootfs path. Verified working.

Environment

• E2B cloud, template built via Python SDK (Template().from_template("code-interpreter-v1")), Debian 13 guest
• ca-certificates-java 20240118, OpenJDK 21
• Observed on builds as of 2026-08-01; root cause traced against current main of this repo