External OCI System Image Materialization for OpenCHAMI

This is a design specification with six requirements and non-goals, and the payload names no existing files, tests or entry points. A newcomer should read the requirements and non-goals first, then ask a maintainer which part, if any, is open for contribution. Done would mean the scenarios of a scoped sub-requirement are met, which this issue does not yet define.

External OCI System Image Materialization for OpenCHAMI

---

Summary

HPC site administrators need a bounded way to qualify an externally produced, boot-ready OCI system image and materialize it into the kernel, initramfs, and SquashFS artifacts consumed by OpenCHAMI. The user supplies the complete source image, boot components, identity, provenance, registry access, destination storage, and functional-group mapping. The capability validates those inputs, preserves source traceability, produces an integrity-verifiable boot bundle, and fails without modifying the source when required inputs are incomplete or incompatible.

The capability is independent of Buildstream and does not make BR-30785 dependent on external OCI ingestion. It is intended as a reusable OpenCHAMI interoperability contribution rather than customer-specific Buildstream code.

---

Problem Statement

Sites may already produce complete operating-system images through external build systems, but OpenCHAMI bare-metal boot requires a distinct kernel, network-capable initramfs, and SquashFS root filesystem published through a compatible object endpoint. Without a supported materialization boundary, each external-image request requires one-off engineering to inspect the OCI image, identify a consistent boot set, flatten its filesystem, create boot artifacts, publish them, and reconstruct provenance.

The latest full-cluster update required two days of downtime. At a minimum monthly RHEL kernel update cadence across the full functional-group matrix, repeating that pattern creates exposure of up to 24 cluster-days per year. Pre-materializing and validating the complete image set before maintenance can reduce the deployment transition target to four hours per monthly event, recovering approximately 22 cluster-days of availability annually.

---

Value Proposition

Customer Value

• Administrators can reuse a complete, externally built OCI system image without rebuilding its operating-system content in Buildstream.
• Every materialized boot bundle remains traceable to an immutable source OCI digest.
• Input deficiencies are reported before maintenance rather than discovered during the deployment window.
• The full functional-group artifact set can be staged and validated before a monthly kernel-update event, targeting a four-hour deployment transition instead of a two-day outage.

---

User Personas

• External Image Producer: Builds and publishes a complete boot-ready OCI system image and supplies provenance, architecture, and authorization to consume it.
• HPC Site Administrator: Supplies source and destination access, functional-group mapping, release identity, and any required kernel selection, then initiates materialization.
• OpenCHAMI Operator: Reviews the resulting boot bundle and mapping, performs destination canary boot validation, and owns provisioning-time site configuration.

---

Use Cases

Use Case 1: Qualify an External OCI System Image

Actor: HPC Site Administrator
Context: An external build system has published an immutable OCI image intended for an OpenCHAMI-managed functional group.
Action: The administrator provides the OCI digest, architecture, registry access, functional-group identity, kernel selection when needed, provenance, and destination information.
Result: The system either confirms one complete, internally consistent boot-ready input set or rejects the request with actionable missing or conflicting inputs.

Use Case 2: Materialize and Publish a Boot Bundle

Actor: HPC Site Administrator
Context: The external OCI image has passed input qualification.
Action: The administrator initiates materialization for the selected architecture, functional group, release, and destination prefix.
Result: The system produces and publishes vmlinuz, initramfs.img, rootfs.squashfs, checksums, provenance, and a compatible artifact mapping without modifying the source OCI image.

Use Case 3: Qualify the Materialized Bundle for OpenCHAMI

Actor: OpenCHAMI Operator
Context: A complete materialized bundle is available at the destination endpoint.
Action: The operator verifies integrity and accessibility, registers the destination mapping through the supported local process, and performs a representative canary boot.
Result: The bundle is accepted for the intended architecture and functional group or rejected without changing the immutable source.

---

Scope Definition

Goals (Execution Outcomes)

• Goal 1: Define and enforce a complete input requirements for externally produced boot-ready OCI system images.
• Goal 2: Materialize a qualified source into an atomic OpenCHAMI kernel, initramfs, and SquashFS boot bundle without rebuilding or modifying source content.
• Goal 3: Preserve source digest, architecture, functional-group, release, checksum, and destination traceability.
• Goal 4: Publish the bundle to a compatible S3-style endpoint and verify integrity and object accessibility before handoff.
• Goal 5: Fail safely when the source is incomplete, ambiguous, incompatible, unauthorized, or inaccessible.
• Goal 6: Provide reusable, site-neutral documentation and contribution artifacts suitable for upstream OpenCHAMI review.

Non-Goals (Scope Boundaries)

• Non-Goal 1: Installing packages, kernels, modules, firmware, system services, cloud-init, or networking support into the source image.
• Non-Goal 2: Generating, repairing, or modifying an initramfs.
• Non-Goal 3: Converting a conventional application container into a bootable system image.
• Non-Goal 4: Resolving package dependencies, certifying drivers, or validating application workloads.
• Non-Goal 5: Vulnerability remediation, SBOM generation, signing the source image, or replacing the external producer's security process.
• Non-Goal 6: Automatic Buildstream integration, Buildstream catalog modification, or making BR-30785 depend on this capability.
• Non-Goal 7: Continuous registry or S3 synchronization, artifact replication, or destination lifecycle management.
• Non-Goal 8: Reconstructing the original OCI layers, manifest history, runtime configuration, or digest from the materialized SquashFS.

---

Requirements

Requirement: Complete External Image Input Requirements [Req 1]

The system must accept only an immutable, authorized OCI system image accompanied by the complete information required to identify, access, validate, materialize, and publish one architecture-specific boot bundle.

Relates to: UC-1 — Qualify an External OCI System Image

Scenario: Complete input accepted

• Given an OCI image pinned by digest, approved registry access and trust, selected architecture, full-system root filesystem, kernel selection where needed, matching modules, existing network-capable initramfs, source provenance, functional-group mapping, release identity, and destination access,
• When the administrator requests qualification,
• Then the system records the immutable source identity and proceeds to boot-component validation.

Scenario: Missing or mutable input rejected

• Given a mutable tag without an immutable digest, missing credentials or trust, absent functional-group or release identity, or missing destination information,
• When qualification runs,
• Then the system rejects the request before materialization and identifies each missing or mutable input.

Requirement: Boot-Readiness Validation [Req 2]

The system must verify that the selected architecture has one complete, mutually consistent kernel, module tree, network-capable initramfs, and full-system root filesystem suitable for the OpenCHAMI live-boot format.

Relates to: UC-1 — Qualify an External OCI System Image

Scenario: One valid boot set

• Given a source containing one selected kernel, its matching module tree, and a matching network-capable initramfs,
• When boot-readiness validation runs,
• Then the system records the selected versions and confirms the source is eligible for materialization.

Scenario: Ambiguous or incomplete boot set

• Given zero valid boot sets, multiple valid kernels without an explicit selection, mismatched modules, or an initramfs that does not satisfy the declared live-network boot requirements,
• When validation runs,
• Then the system fails with actionable details and does not install, generate, or repair missing components.

Requirement: Deterministic Boot-Artifact Materialization [Req 3]

The system must flatten the qualified architecture-specific OCI filesystem and produce one atomic boot bundle containing the selected kernel as vmlinuz, the selected initramfs as initramfs.img, and the root filesystem as rootfs.squashfs, without modifying the immutable source.

Relates to: UC-2 — Materialize and Publish a Boot Bundle

Scenario: Successful materialization

• Given a qualified source and a new immutable release identity,
• When materialization runs,
• Then the system produces the three required artifacts from the selected source and records their sizes and SHA-256 checksums.

Scenario: Materialization failure

• Given insufficient temporary storage, a mount or extraction failure, or a SquashFS creation failure,
• When materialization cannot complete,
• Then the system reports the failed phase, removes or quarantines incomplete temporary output, and leaves the source OCI image unchanged.

Requirement: Atomic Publication and Compatible Mapping [Req 4]

The system must publish a complete bundle to a new immutable destination prefix and generate site-neutral provenance plus an OpenCHAMI/Omnia-compatible mapping that identifies the destination kernel, initramfs, and root filesystem objects.

Relates to: UC-2 — Materialize and Publish a Boot Bundle

Scenario: Complete publication

• Given a successful materialization and authorized destination access,
• When publication completes,
• Then the destination contains all three artifacts, SHA256SUMS, provenance, and the compatible mapping under one release prefix.

Scenario: Partial publication prevented

• Given an existing destination release, insufficient authorization, or a failure during upload,
• When publication cannot complete atomically,
• Then the system does not mark the release ready, reports the incomplete objects, and preserves the prior accepted destination state.

Requirement: Integrity, Provenance, and Handoff Validation [Req 5]

The system must preserve an evidence chain from source OCI digest to destination artifact checksums and verify the destination access behavior required by the target OpenCHAMI Boot Service before declaring materialization successful.

Relates to: UC-3 — Qualify the Materialized Bundle for OpenCHAMI

Scenario: Verified handoff

• Given a complete published bundle,
• When destination validation runs,
• Then artifact checksums match, required object HEAD and bounded GET behavior succeeds, provenance contains no credentials, and the mapping identifies one atomic bundle.

Scenario: Integrity or access failure

• Given a checksum mismatch, inaccessible object, certificate or policy failure, or mapping inconsistency,
• When handoff validation runs,
• Then the release is rejected for use, the failure is reported, and neither the source nor a previously accepted destination release is altered.

Requirement: Representative Boot Qualification [Req 6]

A materialized bundle must support a representative canary boot on compatible target hardware before it is accepted for the intended functional group.

Relates to: UC-3 — Qualify the Materialized Bundle for OpenCHAMI

Scenario: Canary boot accepted

• Given a verified bundle, destination-local provisioning mapping, and compatible target hardware,
• When the OpenCHAMI operator performs the canary boot,
• Then the node loads the published kernel and initramfs, mounts the SquashFS root filesystem, and reaches the destination's defined provisioning completion state.

Scenario: Canary boot rejected

• Given a node that fails kernel, initramfs, network, rootfs, or provisioning completion,
• When the canary result is evaluated,
• Then the bundle remains unaccepted, diagnostic evidence identifies the failed stage, and the operator returns remediation responsibility to the external image producer when the source inputs are deficient.

---

Usability

• Qualification results must identify every missing, conflicting, or ambiguous input before materialization starts.
• An administrator must be able to review source identity, selected architecture, kernel choice, functional group, release, destination, and expected outputs before confirmation.
• The capability must distinguish validation failure, materialization failure, publication failure, and destination boot failure.

---

Reliability

• Source OCI content is never modified.
• A failed run must not mark a partial destination release as ready.
• Temporary mounts and working data are cleaned or quarantined after success and failure.
• Retrying the same immutable source and inputs must not silently overwrite an accepted release.
• Previously accepted destination releases remain available when a new materialization fails.

---

Security

• Registry and destination credentials must be supplied through approved credential-management mechanisms and must not appear in catalogs, logs, provenance, mappings, or command examples.
• The source must be pinned by immutable digest and authorized for use.
• Registry and destination certificate trust must be validated according to site policy.
• Materialization must preserve filesystem ownership, permissions, extended attributes, and capabilities required by the source system image.
• Provenance must identify source and output integrity without exposing internal account identifiers or sensitive values.
• Untrusted source content must be processed within an approved isolation boundary.

---

Performance

Measure	Target	Rationale
Qualification completion	Before temporary rootfs materialization begins	Detect deficient inputs before expensive processing
Monthly full-cluster staging	Complete before the approved maintenance window	Remove image preparation from cluster downtime
Deployment transition objective	Four hours or less for the full functional-group set	Recover approximately 22 cluster-days per year versus the two-day baseline

Materialization duration depends on source image size, architecture count, storage throughput, and compression resources. The capability records observed duration and artifact sizes for capacity planning rather than imposing an unsupported universal throughput SLA.

---

Telemetry

The capability records operational metrics for qualification duration, materialization duration, artifact size, publication duration, success or failure phase, and cleanup outcome. Operators use these records to plan monthly full-cluster staging and identify recurring source or destination failures. No separate telemetry platform is required.

---

APIs

N/A — This specification does not require a public API.
Rationale: The capability defines observable input, materialization, publication, and evidence interfaces; implementation may expose them through a supported CLI, service, or upstream OpenCHAMI interface.

---

Documentation

• External OCI input-requirements reference.
• Boot-readiness validation and kernel-selection guide.
• Materialization, publication, cleanup, and retry procedure.
• Provenance and checksum schema.
• Destination mapping and OpenCHAMI handoff guide.
• Canary boot qualification checklist.
• Upstream OpenCHAMI contribution documentation and maintainer handoff notes.

---

Dependencies

• OCI-Compatible Source Registry: Supplies immutable architecture-specific system-image content and authorized digest access.
• Complete External System Image: Supplies the full root filesystem, kernel, matching modules, network-capable initramfs, and source provenance.
• Compatible Materialization Environment: Provides sufficient architecture support, temporary storage, filesystem-metadata preservation, and SquashFS capability.
• S3-Compatible Destination: Provides an immutable destination prefix and the object-access behavior required by the target Boot Service.
• OpenCHAMI Boot Artifact Format: Defines the kernel, initramfs, rootfs, and destination mapping consumed during provisioning.
• Compatible Target Hardware and Provisioning Environment: Required for representative canary boot acceptance.
• OpenCHAMI Upstream Review: Required to accept the reusable contribution into the upstream project; upstream acceptance timing does not make BR-30785 dependent on this work.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Application Container Supplied as System Image	Technical	High	Medium	Enforce the complete input and boot-readiness requirements before materialization.
Kernel, Module, or Initramfs Mismatch	Technical	High	Medium	Verify one selected internally consistent boot set; fail rather than generate or repair components.
Untrusted External Content	Security	High	Medium	Require immutable digest, authorization, approved isolation, certificate trust, and source provenance.
Filesystem Metadata Loss	Technical	High	Low	Validate ownership, modes, extended attributes, and capabilities before publishing the bundle.
Partial Destination Release	Operational	High	Medium	Publish under a new immutable prefix and mark ready only after complete integrity and access validation.
Unsupported Target Hardware	Technical	Medium	Medium	Require representative canary boot on compatible hardware before functional-group acceptance.
Scope Expansion into Image Building	Delivery	High	Medium	Keep package installation, initramfs generation, driver remediation, and dependency resolution explicit Non-Goals.
Upstream Contribution Delay	Operational	Medium	Medium	Keep the capability site-neutral and independently testable while coordinating contribution review with OpenCHAMI maintainers.
Downtime Savings Not Realized	Business	High	Medium	Measure preparation and deployment windows during monthly full-cluster updates and compare them with the two-day baseline.

---

Acceptance Criteria

• AC-001: A complete external OCI system image pinned by digest is accepted only when registry access, trust, architecture, functional-group mapping, release identity, destination access, provenance, and the complete boot set are present. [Req 1, Req 2]
• AC-002: Mutable, incomplete, ambiguous, unauthorized, or boot-incompatible sources are rejected before materialization with actionable validation results. [Req 1, Req 2]
• AC-003: A qualified source produces vmlinuz, initramfs.img, and rootfs.squashfs without modifying the immutable source OCI image. [Req 3]
• AC-004: The destination release contains the three artifacts, SHA256SUMS, site-neutral provenance, and a compatible atomic artifact mapping under a new immutable prefix. [Req 4]
• AC-005: Destination checksums match the materialized source artifacts, required object access succeeds, and no credentials appear in provenance or mappings. [Req 5]
• AC-006: A failed materialization or publication reports the failed phase, cleans or quarantines incomplete state, leaves the source unchanged, and does not replace a previously accepted release. [Req 3, Req 4, Req 5]
• AC-007: A representative compatible node successfully boots the materialized bundle through OpenCHAMI and reaches the defined provisioning completion state before the bundle is accepted for the functional group. [Req 6]
• AC-008: For a monthly full-cluster RHEL kernel update exercise, all functional-group bundles are materialized and staged before maintenance, and the measured deployment transition completes within the four-hour objective. [Req 1, Req 3, Req 4, Req 6]

---

Scope & Sequencing

The first increment defines the complete input requirements, validates one architecture-specific system image per run, materializes the atomic boot bundle, publishes it to a compatible destination, and validates a representative canary boot. Multi-architecture OCI indexes are handled by explicitly selecting and qualifying each architecture independently. Upstream OpenCHAMI contribution preparation can proceed alongside local validation, but BR-30785 and Buildstream remain independent.

---

Assumptions / Open Questions

Assumptions

• A1: The external producer supplies a complete boot-ready system image rather than an application container.
• A2: The customer can provide immutable source identity, access, trust, architecture, boot-component, functional-group, release, destination, and provenance inputs.
• A3: Monthly RHEL kernel updates require materialization across the full supported functional-group matrix.
• A4: The observed two-day update outage is a valid baseline, and four hours is the target deployment-transition window after pre-materialization.
• A5: The target OpenCHAMI environment supports kernel, initramfs, and SquashFS boot artifacts through a compatible object endpoint.

Open Questions

• OQ-1: Which OpenCHAMI repository and maintainer group will own upstream review and long-term maintenance of the contribution?
• OQ-2: Which architecture and operating-system combinations will form the initial upstream conformance test matrix?

---