Diskful Boot Option with OpenCHAMI

Start with omnia_config.yml and how functional groups are classified as Compute, K8s Worker, Manager or Storage, since the proposed boot_mode: diskful | diskless toggle has to be checked against those roles. The issue depends on upstream OpenCHAMI schema changes, so confirm with maintainers what can be done in this repo first. Done means AC-004 holds: diskful on a Compute or K8s Worker group fails the playbook with a clear error and modifies nothing.

Diskful Boot Option with OpenCHAMI

---

Summary

Traditional HPC deployments heavily utilize "diskless" (stateless) booting via PXE to run compute nodes entirely from RAM, simplifying management and updates. However, the convergence of HPC and AI—specifically the introduction of Kubernetes-based AI workflows (e.g., RHEL AI) and massive containerized workloads—demands persistent local storage on certain node roles. This capability enhances Omnia's bare-metal provisioning stack (OpenCHAMI) to support a native "Diskful Boot" option exclusively for explicitly designated Manager or Storage nodes (e.g., K8s control plane, Proxmox hosts, dedicated storage servers). Compute nodes and K8s worker nodes remain strictly stateless — they must continue to PXE boot into RAM-disk images with no persistent local OS state. This will allow administrators to instruct OpenCHAMI to partition local NVMe/SSD drives and permanently install the host operating system on eligible node roles, providing the local state necessary to cache multi-gigabyte AI container images and provide high-speed local scratch space.

---

Problem Statement

As Omnia expands from pure HPC batch processing (Slurm) into converged AI (Kubernetes) and container-heavy architectures, certain management and storage node roles require persistent local disk mounts to host etcd state, container registries, or large localized caching layers. Compute nodes and K8s worker nodes must remain entirely RAM-disk backed (stateless). Without a supported diskful provisioning workflow for eligible node roles, administrators are forced to manually image these nodes outside of the OpenCHAMI automation loop, breaking Day 1 cluster orchestration and destroying the unified infrastructure management experience.

---

Value Proposition

Customer Value

• Converged Provisioning: Provides a single control plane (OpenCHAMI) capable of simultaneously managing both traditional stateless Slurm nodes and diskful Kubernetes/AI nodes.
• Performance: Leverages high-speed local NVMe drives on designated Manager and Storage nodes for caching AI container images and deep learning scratch space without saturating the InfiniBand/Ethernet fabric pulling images from network storage.

---

User Personas

• AI/HPC Infrastructure Architect: Designs the cluster, deciding which logical roles (Functional Groups) require stateful local storage vs. diskless provisioning.
• Omnia Infrastructure Administrator (Infra Admin): Uses the Omnia config and OpenCHAMI to execute the bare-metal provisioning rules against specific BMC MAC addresses.

---

Use Cases

Use Case 1: Provisioning Stateful AI Nodes

Actor: Omnia Provisioning Engine (OpenCHAMI)
Context: Administrator defines a host group of designated Manager or Storage nodes (e.g., K8s control plane, dedicated storage servers) intended for persistent local state. Compute nodes and K8s worker nodes are excluded — they remain stateless and RAM-disk backed.
Action: Omnia triggers OpenCHAMI to boot the eligible nodes via PXE, stream the pre-compiled Golden Image to the local NVMe storage, and execute cloud-init on first-boot to configure partitions and network.
Result: The designated server boots from its local disk on subsequent restarts, retaining state across power cycles. Compute and worker nodes continue to PXE boot statelessly.

---

Technical Context & Implementation Mechanisms

To implement diskful booting via OpenCHAMI, Omnia will leverage its existing mature Golden Image build workflow and cloud-init capabilities rather than reinventing the wheel with legacy installers:

1. Image-Based Provisioning (Primary Path): Streaming a pre-compiled raw disk image (generated by Omnia's existing Golden Image pipeline) directly to the local block storage (NVMe/SSD) using a micro-OS or initramfs. This is significantly faster than traditional installers and ensures diskful nodes are perfectly identical to diskless clones.
2. cloud-init / Ignition Configuration: Once the Golden Image is written to the block device, standard cloud-init or Ignition processes will execute on first-boot to dynamically configure local state, expand custom partitions (like /var/lib/containers), inject network settings, and configure the GRUB2 EFI bootloader. The cloud-init runcmd module will execute grub2-install and efibootmgr to register the EFI bootloader with the motherboard's UEFI NVRAM, ensuring the node can successfully boot from the local disk on subsequent restarts.

Note: Legacy file-based installation methods (like Kickstart or Preseed) are considered out-of-scope for this capability, as they would duplicate effort, bypass the Golden Image pipeline, and increase Day 1 provisioning time.

Omnia engineering will need to collaborate with the OpenCHAMI community to ensure OpenCHAMI's upstream schema natively supports streaming these Golden Images to block devices and passing the necessary cloud-init user-data payloads.

---

Scope Definition

Goals (Execution Outcomes)

• Enhance OpenCHAMI integration to support diskful OS installation (writing the image permanently to local block storage) only for explicitly designated Manager or Storage node roles.
• Expose a simple toggle in omnia_config.yml (e.g., boot_mode: diskful | diskless) per node group. The playbook must reject diskful for any functional group classified as Compute or K8s Worker, enforcing the stateless constraint.
• Support automated disk wiping/repartitioning during the Day 1 provisioning cycle for eligible node roles only.

Non-Goals (Scope Boundaries)

• This capability does not replace the existing diskless booting method; both must be supported.
• This does not cover managing or provisioning distributed filesystems (like Ceph or Lustre) over those local disks; it strictly focuses on the local OS and scratch mount.

---

Requirements

Requirement 1: Configurable Boot Mode [Req 1]

The system must provide a configuration mechanism to map individual nodes or logical groups to either a diskful or diskless boot profile. The diskful mode is only permitted on explicitly designated Manager or Storage node roles (e.g., K8s control plane, dedicated storage servers). Compute nodes, K8s worker nodes, and Slurm compute/login nodes must remain diskless (stateless, RAM-disk backed). The system must reject any configuration that assigns diskful to a disallowed functional group.

Requirement 2: OpenCHAMI Diskful Pipeline [Req 2]

The OpenCHAMI integration must support writing an OS image permanently to the local block device, configuring the host to boot from that local disk on subsequent reboots, and ensuring the GRUB2 EFI bootloader is correctly installed and registered with the UEFI NVRAM via cloud-init.

Requirement 3: Automated Partitioning [Req 3]

The diskful provisioning process must allow administrators to define custom partitioning schemes (e.g., allocating a large /var/lib/containers partition).

---

Usability

The Omnia inventory file (omnia_config.yml) must clearly expose the boot option without requiring the administrator to manually write complex PXE boot scripts.

---

Reliability

Failure Mode	System Behavior
Target Disk Missing	If the diskful target (e.g., /dev/sda or nvme0n1) is missing or unresponsive, OpenCHAMI must fail the deployment for that node gracefully and log an error to the central OIM node.

---

Security

• Diskful nodes must support Secure Boot integration, ensuring the locally written kernel and bootloader are signed.

---

Documentation

• Provisioning Guide: Must include a dedicated section contrasting Diskless vs. Diskful architectures and best practices for configuring OpenCHAMI to handle hybrid deployments.

---

Dependencies

• OpenCHAMI Upstream: Deep collaboration with the OpenCHAMI open-source project is required. Current OpenCHAMI architectures are optimized for stateless/diskless booting. We will require upstream feature support and schema changes in OpenCHAMI APIs to standardize diskful profiles, define target block devices, handle partitioning mechanisms, and automate post-provisioning boot order changes.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Wear and Tear	Operational	Low	Medium	Persistent writing to local OS drives may cause early SSD failure compared to RAM-disk HPC. Mitigate by documenting best practices for enterprise-grade write-intensive SSDs.

---

Acceptance Criteria

• AC-001: An administrator can successfully provision a designated Manager or Storage node using OpenCHAMI where the OS is permanently installed to the local hard drive, and the GRUB2 EFI bootloader is correctly installed and registered. [Req 1]
• AC-002: Upon a soft reboot (reboot), the diskful node boots directly from the local disk without needing to pull the OS image over the network again. [Req 1]
• AC-003: The same Omnia cluster can simultaneously manage a group of diskless nodes (Compute, K8s Workers) and a group of diskful nodes (Manager, Storage) without conflicts. [Req 1]
• AC-004: Setting boot_mode: diskful on a functional group classified as Compute or K8s Worker causes the provisioning playbook to fail with a clear error rejecting the configuration. The node is not modified. [Req 1]

---

Scope & Sequencing

N/A — This is a single delivery increment with no phased approach or specific sequencing dependencies.
Rationale: Follows foundational OpenCHAMI deployment.

Assumptions / Open Questions

Assumptions

• A1: Required underlying hardware and OS capabilities are present.

Open Questions

• None.