Track release download bugs: missing initramfs provenance

Start in scripts/nvx_tools/release.py at _install_release_archive and compare its provenance_sources with the three files copied by package_release. Read the runtime-layout and source-inclusive installation tests in scripts/test_nvx_tools.py, then extend them to cover installing and replacing initramfs provenance. Done means both tests verify all three provenance files are installed and an older initramfs provenance file is replaced.

Summary

A focused, read-only bug hunt found one confirmed Medium-severity bug in release download/installation: the package includes initramfs build provenance, but the installer never restores it.

Revision: latest origin/dev fetched on 2026-10-11, 6020d5d49e2e4839feb46cbadbc8db7e8cf334e2. All code permalinks below use this SHA.

Trigger and impact

Run the documented download command into a checkout with no locally built initramfs provenance, or upgrade a runtime that already has provenance from an older build.

• Expected: installation copies all packaged runtime provenance, including the initramfs provenance, and replaces existing copies with the ones belonging to the installed artifacts.
• Actual: installation completes normally, copies the binary/guest artifacts and the OpenVMM/kernel provenance, but ignores the packaged initramfs provenance. A fresh runtime lacks the file; an upgrade retains the previous file.
• Impact: a successful download does not produce the artifact inventory required by runtime provenance validation, release repackaging, or adversarial executor preflight. Fresh installs fail with initramfs build provenance not found / initramfs provenance not found; upgrades can leave provenance that does not describe the newly installed initramfs.

Trace and permalinks

1. command_download calls download_checkout_release, which installs the archive before reporting success.
2. package_release copies all three provenance files into the package's provenance/ directory, including InitramfsBuildConstants.PROVENANCE_NAME.
3. _install_release_archive builds provenance_sources from only OpenVMMBuildConstants.PROVENANCE_NAME and KernelBuildConstants.PROVENANCE_NAME. Its final copy loop therefore never installs initramfs provenance.
4. GUEST_ARTIFACT_NAMES also excludes that file, so the guest copy loop does not compensate.
5. _require_runtime_provenance_paths requires all three files. Both validate_runtime_artifact_provenance and release packaging use this requirement.
6. Adversarial executor _artifact_metadata independently requires the initramfs provenance before validating artifacts. Executor preflight calls it before the baseline canary boot.

Smallest reproduction

Run the following standalone Python script from the root of a checkout or git archive export of the SHA above. Save the script outside the checkout. It uses only the standard library and repository modules.

The fixture deliberately uses inert payloads: it tests the actual checksummed archive creation/extraction and runtime installation paths without executing OpenVMM. Only the two destination-directory constants are redirected to a temporary directory; no installer, validator, copy function, or failure is mocked.

import json
import os
import sys
import tempfile
from pathlib import Path
from unittest.mock import patch

sys.path.insert(0, str(Path.cwd() / "scripts"))

from nvx_tools import adversarial_executor, common, release
from nvx_tools.build_constants import (
    BuildConstants,
    InitramfsBuildConstants,
    KernelBuildConstants,
    OpenVMMBuildConstants,
    ReleaseBuildConstants,
)

with tempfile.TemporaryDirectory(prefix="nvx-provenance-repro-") as temporary:
    root = Path(temporary)
    package = root / "nvx-fixture"
    binary_name = "openvmm.exe" if os.name == "nt" else "openvmm"
    payloads = {
        f"bin/{binary_name}": b"inert binary fixture",
        **{
            f"guest/{name}": f"guest fixture: {name}".encode()
            for name in ReleaseBuildConstants.GUEST_ARTIFACT_NAMES
        },
        **{
            f"provenance/{name}": json.dumps({"fixture": name}).encode()
            for name in (
                OpenVMMBuildConstants.PROVENANCE_NAME,
                KernelBuildConstants.PROVENANCE_NAME,
                InitramfsBuildConstants.PROVENANCE_NAME,
            )
        },
        "SOURCE-MANIFEST.json": json.dumps(
            {"linux": {}, "alpine": {}, "ubuntu": {}, "azurelinux": {}}
        ).encode(),
    }
    for name, contents in payloads.items():
        path = package / name
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_bytes(contents)
    common.write_sha256_sums(package)
    extension = ".zip" if os.name == "nt" else ".tar.gz"
    archive = root / f"nvx-fixture{extension}"
    release.create_release_archive(package, archive)

    build_dir = root / "runtime" / "build"
    expected = payloads[
        f"provenance/{InitramfsBuildConstants.PROVENANCE_NAME}"
    ]
    with (
        patch.object(BuildConstants, "BUILD_DIR", build_dir),
        patch.object(
            OpenVMMBuildConstants, "DIRECTORY", root / "runtime" / "openvmm"
        ),
    ):
        release._install_release_archive(archive)
        missing = common.artifact_path(InitramfsBuildConstants.PROVENANCE_NAME)
        other_provenance = (
            OpenVMMBuildConstants.PROVENANCE_NAME,
            KernelBuildConstants.PROVENANCE_NAME,
        )
        assert common.openvmm_binary_path().is_file()
        assert all(
            common.artifact_path(name).is_file()
            for name in ReleaseBuildConstants.GUEST_ARTIFACT_NAMES
        )
        assert all(
            common.artifact_path(name).is_file() for name in other_provenance
        )
        print("INSTALL_COMPLETED=True")
        print("PACKAGED_INITRAMFS_PROVENANCE=True")
        print(f"INSTALLED_INITRAMFS_PROVENANCE={missing.is_file()}")
        assert not missing.exists(), "The missing-provenance bug did not reproduce"
        for label, check in (
            ("RUNTIME_PROVENANCE_CHECK", release._require_runtime_provenance_paths),
            ("ADVERSARIAL_PREFLIGHT", adversarial_executor._artifact_metadata),
        ):
            try:
                check()
            except common.ScriptError as error:
                print(f"{label}: {error}")
            else:
                raise AssertionError(f"{label} unexpectedly succeeded")

        missing.write_bytes(expected)
        assert len(release._require_runtime_provenance_paths()) == 3
        print("CONTROL_REQUIRED_PROVENANCE_INVENTORY=PASS")
        missing.write_bytes(b"previous release provenance")
        release._install_release_archive(archive)
        assert missing.read_bytes() == b"previous release provenance"
        assert missing.read_bytes() != expected
        print("REINSTALL_LEAVES_PREVIOUS_INITRAMFS_PROVENANCE=True")

Observed output (temporary-directory paths normalized; reproduction exits 0 because it asserts the buggy behavior):

>> archived <temp>\nvx-fixture as <temp>\nvx-fixture.zip
INSTALL_COMPLETED=True
PACKAGED_INITRAMFS_PROVENANCE=True
INSTALLED_INITRAMFS_PROVENANCE=False
RUNTIME_PROVENANCE_CHECK: initramfs build provenance not found: <temp>\runtime\build\initramfs.provenance.json
ADVERSARIAL_PREFLIGHT: initramfs provenance not found: <temp>\runtime\build\initramfs.provenance.json
CONTROL_REQUIRED_PROVENANCE_INVENTORY=PASS
REINSTALL_LEAVES_PREVIOUS_INITRAMFS_PROVENANCE=True

The control copies only the omitted package member and verifies that the required-provenance inventory then succeeds. It does not claim that an inert fixture can boot or pass semantic provenance validation.

Suggested fix

Include InitramfsBuildConstants.PROVENANCE_NAME in the installer's provenance_sources, so it is validated and installed with the other provenance files. Keep the package/installer provenance inventories aligned.

Extend the existing runtime-layout installation test and source-inclusive installation test to package and assert all three provenance files, including replacement of an old initramfs provenance file. Their current fixtures contain only OpenVMM/kernel provenance, which explains why they pass despite this omission.

What was explored

• Read repository guidance and the authoritative quality action and CLI validation action.
• Searched repo:microsoft/nvx label:bug-hunt, including open and closed issues. Open hunts cover CI required statuses (#496), guest artifact cache keys (#490), live /run shares (#486), and managed control-pipe timeouts (#464), not release download/installation. The closed release source-collection hunt (#374) concerns a separate collection pipeline; no existing downloader/install tracker was found.
• Focused on release selection, archive/checksum verification, runtime installation, and the consumers of installed provenance.
• Traced the omission through the CLI caller, package producer, installer, downstream consumers, and existing tests.
• Stopped investigating immediately after this first confirmed bug, well within the approximately 45-minute budget.

Checked, not filed

• First-parent release selection, draft-release handling, and authenticated/public download fallback: existing focused tests pass; no new confirmed bug.
• Archive verification and staged package publication/rollback: existing focused tests pass; no new confirmed bug.
• Sequential per-file runtime installation after a later filesystem write failure: reviewed but not reproduced before the one-bug stop; unconfirmed, not filed.
• The earlier Alpine recipe-symlink source-collection issue is already fixed and was not refiled.

Validation and limits

• Windows, Python 3.14.7; exact latest-dev source exported with git archive into session storage outside the worktree.
• 43 tests passed:

  python -m unittest scripts.test_nvx_tools.ReleaseTests scripts.test_nvx_tools.DownloadTests -v
  Ran 43 tests in 8.191s
  OK
  

• The standalone reproduction passed its assertions using a real ZIP archive created and verified by the repository's archive path, including fresh installation, the missing-file control, and stale-provenance reinstall.
• Required python scripts\nvx.py verify was attempted first and exited 1 because the OpenVMM submodule is uninitialized (initialized OpenVMM submodule not found: ...\openvmm\Cargo.toml). This is an environment prerequisite, not a filed bug.
• No real release was downloaded, no build or hypervisor operation was performed, and no full adversarial campaign was run. Linux TAR behavior is implied by the shared installer inventory but was not executed on a Linux host.
• No repository files were modified and no commits were created; the source worktree and uninitialized submodule were preserved.