Track Ubuntu native build bugs: source changes mislabel stale initramfs

Start with build_initramfs in scripts/nvx_tools/build.py and the Ubuntu manifest and input hash functions in scripts/nvx_tools/ubuntu.py; compare the Alpine input mutation handling and kernel regression test in scripts/test_nvx_tools.py. Add a test that edits a customization source during packing, alongside an unchanged input control. Done when the changed input causes the build to fail and removes its artifact and manifest, while the control succeeds.

Hunt scope and revision

• Repository: microsoft/nvx; hunt started 2026-10-12 at approximately 04:00 UTC.
• Latest origin/dev fetched for this hunt: a5df986e5e606a82add90646fb6a3f9de0a2ec69. All source permalinks below use this SHA.
• Focus: guest artifact build preparation, incremental reuse, and source-input provenance. Existing open and closed bug-hunt issues were searched; none tracks Ubuntu native build provenance. Host telemetry was excluded after discovering that #499 had already explored it.
• The worktree is at c29f3d1b032938503369ba9c14a0063241317627. Its executable code and repository guidance are identical to the pinned latest origin/dev: the only differences are documentation and performance CSVs.
• Investigation stopped at the first confirmed bug. No repository files were edited and no commits were created; reproduction fixtures lived in automatically cleaned external temporary directories.

Confirmed finding: concurrent source changes mislabel a stale Ubuntu initramfs

Severity: Medium. A normal source edit during a native Ubuntu initramfs build can produce incorrect build provenance and let a stale guest artifact pass the release guest-input freshness gate. This is a build/release correctness issue, not a claimed exploit.

Trigger

Run the native Ubuntu initramfs build, and change a customization source such as guest/ubuntu/nvx-bashrc after it has been installed into the prepared root but before the post-pack manifest digest is computed. The build can spend time compiling helpers and packing the archive in this window.

Trace and permalinks

1. The public build-initramfs --guest ubuntu command reaches the native builder through command_build_initramfs.
2. build_initramfs captures pre-build provenance inputs and checks for changes only for Alpine. For Ubuntu, it installs and packs the root first, then computes ubuntu.converter_input_sha256(ubuntu.customization_files()) for the manifest.
3. _install_guest_files has already copied the old Ubuntu shell customization and recorded its old helper source hash.
4. _write_ubuntu_manifest records the completed artifact's hash alongside the later input hash. The customization input hash includes the now-edited customization source.
5. The release _guest_release_inputs gate accepts this artifact: both the output hash and the falsely current input_sha256 match. It does not cross-check the helper's older source hash against the current customization file.

Expected versus actual

Expected: Detect that the build's source inputs changed, fail explicitly, and remove or avoid publishing the artifact and its manifest. This is already the pattern for Alpine initramfs builds and native kernel input mutation.

Actual: The Ubuntu build returns success. Its output contains the old customization; the manifest advertises the current/new input digest while its helper metadata still records the old source hash. The actual release guest-input validator accepts the result.

Minimal controlled reproduction

Run from the repository root at the pinned revision. Save this script outside the checkout, then run python -B <script-path>.

The reproduction exercises the actual native builder, shell-file installation, input hashing, manifest generation, and release guest-input validator. It substitutes only the Linux platform guard, downloaded root preparation, external compiler/archive tools, and package-record fixture. Its archive stand-in contains the installed shell script's bytes; it is not a bootable initramfs. Editing an external temporary source copy in the pack callback makes the race deterministic. The unchanged-source control runs first.

import json
import shutil
import sys
import tempfile
from pathlib import Path
from unittest.mock import patch

REPO = Path.cwd()
sys.path.insert(0, str(REPO / "scripts"))

from nvx_tools import build, common, release, ubuntu
from nvx_tools.build_config import InitramfsBuildConfig
from nvx_tools.build_constants import (
    BuildConstants,
    ReleaseBuildConstants,
    UbuntuBuildConstants,
)


def reproduce(mutate):
    with tempfile.TemporaryDirectory(prefix="nvx-ubuntu-input-hunt-") as temporary:
        root = Path(temporary)
        for relative in ("guest/common", "guest/ubuntu"):
            parts = relative.split("/")
            shutil.copytree(REPO.joinpath(*parts), root.joinpath(*parts))
        lock = root / "ubuntu" / "packages.lock.json"
        lock.parent.mkdir()
        shutil.copyfile(UbuntuBuildConstants.PACKAGE_LOCK, lock)
        source = root / "guest" / "ubuntu" / "nvx-bashrc"
        source.write_bytes(source.read_bytes() + b'\nexport NVX_HUNT_STATE="old"\n')

        guest_root = root / "work" / "root"
        (guest_root / "sbin").mkdir(parents=True)
        (guest_root / "etc").mkdir()
        artifacts = root / "artifacts"
        artifacts.mkdir()
        output = artifacts / UbuntuBuildConstants.INITRAMFS_NAME

        def compile_helper(work, source, destination, *args, **kwargs):
            destination.write_bytes(b"fixture for external compiler")
            return {
                "source_sha256": common.sha256_file(source),
                "binary_sha256": common.sha256_file(destination),
            }

        def device_helper(work, destination):
            return compile_helper(
                work, root / "guest" / "common" / "nvx-device-io.c", destination
            )

        def pack(prepared_root, destination):
            installed = prepared_root / "etc" / "nvx-bashrc"
            destination.write_bytes(installed.read_bytes())
            if mutate:
                source.write_bytes(
                    source.read_bytes().replace(
                        b'NVX_HUNT_STATE="old"', b'NVX_HUNT_STATE="new"'
                    )
                )

        with (
            patch.object(BuildConstants, "REPO_ROOT", root),
            patch.object(UbuntuBuildConstants, "PACKAGE_LOCK", lock),
            patch.object(build, "_require_linux"),
            patch.object(build, "_prepare_guest_root", return_value=guest_root),
            patch.object(build, "_build_static_helper", side_effect=compile_helper),
            patch.object(build, "_build_device_io_helper", side_effect=device_helper),
            patch.object(build, "_pack_initramfs", side_effect=pack),
            patch.object(ubuntu, "package_records", return_value=[]),
        ):
            before = ubuntu.converter_input_sha256(ubuntu.customization_files())
            build.build_initramfs(
                InitramfsBuildConfig(
                    guest="ubuntu", work=root / "work", output=output
                )
            )
            after = ubuntu.converter_input_sha256(ubuntu.customization_files())
            manifest = json.loads(
                output.with_name(f"{output.name}.packages.json").read_text(
                    encoding="utf-8"
                )
            )
            assert manifest["input_sha256"] == after
            assert b'NVX_HUNT_STATE="old"' in output.read_bytes()
            if mutate:
                assert before != after
                assert b'NVX_HUNT_STATE="new"' in source.read_bytes()
                assert manifest["helpers"]["nvx-bashrc"]["source_sha256"] != (
                    common.sha256_file(source)
                )
            else:
                assert before == after

            for name in ReleaseBuildConstants.GUEST_ARTIFACT_NAMES:
                artifact = artifacts / name
                if not artifact.exists():
                    artifact.write_bytes(b"unrelated guest-artifact fixture")
            distro = artifacts / UbuntuBuildConstants.DISTRO_NAME
            (artifacts / UbuntuBuildConstants.DISTRO_MANIFEST_NAME).write_text(
                json.dumps(
                    {
                        "format": UbuntuBuildConstants.PACKAGE_MANIFEST_VERSION,
                        "guest": "ubuntu",
                        "artifact": distro.name,
                        "artifact_sha256": common.sha256_file(distro),
                        "input_sha256": after,
                    }
                ),
                encoding="utf-8",
            )
            with patch.object(
                release, "artifact_path", side_effect=artifacts.joinpath
            ):
                release._guest_release_inputs(include_azurelinux=False)

            label = "MUTATION" if mutate else "CONTROL"
            print(f"{label}: build returned successfully")
            print(f"{label}: input digest changed = {before != after}")
            print(f"{label}: artifact contains OLD customization")
            print(f"{label}: manifest matches CURRENT input digest")
            print(f"{label}: release guest-input gate ACCEPTED the artifact")


reproduce(False)
reproduce(True)

Observed output, omitting temporary-path build messages:

CONTROL: build returned successfully
CONTROL: input digest changed = False
CONTROL: artifact contains OLD customization
CONTROL: manifest matches CURRENT input digest
CONTROL: release guest-input gate ACCEPTED the artifact
MUTATION: build returned successfully
MUTATION: input digest changed = True
MUTATION: artifact contains OLD customization
MUTATION: manifest matches CURRENT input digest
MUTATION: release guest-input gate ACCEPTED the artifact

All reproduction assertions pass; the mutation case also asserts that the helper's recorded source SHA differs from the current source SHA.

Suggested fix

Capture the Ubuntu customization input digest before preparing/installing the root; use that captured digest in the manifest, and recompute/compare it after packing before publishing success. If it changed, remove the artifact and its manifest and raise ScriptError, mirroring the Alpine/kernel fail-closed pattern. Add a regression test mutating a customization source during packing, plus the unchanged-source control. Checking helper source hashes at release validation can provide defense in depth, but is not a substitute for a consistent build-input snapshot.

Validation and limits

• python -B -m unittest scripts.test_nvx_tools.BuildTests scripts.test_nvx_tools.BuildConstantsTests -q: 83 tests run, OK.
• The controlled reproduction above completed successfully for both unchanged and changed inputs and exercised the real release guest-input freshness gate.
• Existing Ubuntu manifest test checks artifact/hash binding, and stale-input release test changes the manifest digest directly. Neither tests a source edit during the build followed by a falsely current digest.
• Host: Windows, Python 3.14.7. PYTHONDONTWRITEBYTECODE=1 and -B prevented repository bytecode writes.
• python -B scripts\nvx.py verify failed because the OpenVMM submodule is uninitialized (openvmm\Cargo.toml is absent). It was intentionally not initialized or changed.
• No full Linux-native build, guest boot, complete package publication, or hypervisor test is claimed. The confirmed behavior is the platform-independent provenance ordering and acceptance by the specific release guest-input gate, reproduced with controlled external build-tool substitutions.

Checked, not filed

• Host provenance/storage telemetry: excluded when existing #499 coverage was discovered; no additional finding.
• Kernel source/build fingerprints, configuration mutation guards, native/Docker routing, and Ubuntu extraction/metadata helpers: reviewed with the focused existing test suite; no additional confirmed defect.
• A suspected missing parent for a custom programmatic Alpine initramfs output was not reproduced as a supported CLI failure: the CLI uses the standard build destination. Not filed.
• Ubuntu distro-layer input timing is a related surface, but was not independently reproduced and is not a second confirmed finding.