Rocky 10.x OS support

Start with the base OS playbook and the RHSM registration tasks it runs, plus the ansible_distribution conditionals in the existing RHEL playbooks, and the omnia_config.yml OS selection. Done means a Rocky 10 cluster provisions with BaseOS, AppStream, CRB and EPEL configured and no RHSM errors, a mixed-mode Slurm test job runs, and DKMS builds of the drivers load. This is cross-cutting design work, so confirm the approach with maintainers before writing code.

Rocky 10.x OS support

---

Summary

While Red Hat Enterprise Linux (RHEL) is the gold standard for enterprise environments, its subscription costs are prohibitive for massive, scale-out HPC clusters in the academic and public research sectors. This capability introduces Rocky Linux 10.x as a fully supported, first-class operating system in Omnia. By dynamically mapping package repositories (AppStream, CRB, EPEL) and enabling kernel module compilation (DKMS) for proprietary drivers, Omnia will allow customers to deploy pure Rocky 10 clusters, or "mixed-mode" clusters (RHEL for management nodes, Rocky for compute nodes), drastically reducing software licensing costs.

---

Problem Statement

Customers with tight budgets—particularly national labs and universities with thousands of compute nodes—frequently mandate open-source, binary-compatible alternatives to RHEL.

Currently, Omnia's provisioning logic is tightly coupled to RHEL subscription entitlements. Red Hat provides "baked-in" access to extra repositories and pre-compiled proprietary kernel modules (such as NVIDIA CUDA and Mellanox OFED drivers). If an administrator attempts to deploy Rocky Linux today, Omnia fails because it cannot find the Red Hat CDN endpoints, and hardware drivers fail to install because the Rocky kernel signature does not match the pre-compiled RHEL vendor modules. Without natively automating dynamic repository resolution and driver compilation from source, Omnia forces customers to either pay for large-scale RHEL license deployments or abandon the Omnia software stack entirely.

---

Value Proposition

Customer Value

• Significant Cost Savings: Customers save substantially on annual OS subscription fees across large-scale compute node fleets.
• Mixed-Mode Flexibility: Customers can retain commercial RHEL support for their mission-critical management infrastructure (OIM, Slurm Controllers) while utilizing Rocky Linux for their highly ephemeral compute nodes.

---

User Personas

• Academic HPC Infrastructure Administrator (Infra Admin): Needs to provision thousands of compute nodes reliably on a free, stable OS while still getting the automated deployment benefits of Omnia.
• IT Procurement / Budget Director: Mandates the use of open-source software on scale-out nodes to keep the cluster architecture within budget constraints.

---

Use Cases

Use Case 1: Pure Rocky 10 Deployment

Actor: HPC Admin
Context: Provisioning a cost-sensitive academic cluster.
Action: The admin defines the base OS as Rocky 10 in the Omnia inventory and triggers deployment.
Result: Omnia successfully configures the Rocky package managers, points to the correct open-source repositories (AppStream, CRB), and provisions the entire cluster on Rocky Linux 10.

Use Case 2: Mixed-Mode OS Cluster

Actor: HPC Admin
Context: An enterprise wants official Red Hat support for control planes, but free OS for compute.
Action: The admin maps the OIM and Service nodes to RHEL 10, and the Slurm compute nodes to Rocky 10.
Result: Omnia successfully handles the divergent OS logic. The control plane runs RHEL, the workers run Rocky, and the Slurm cluster successfully executes jobs across the boundary.

Use Case 3: Proprietary Driver Compilation

Actor: System (Automated)
Context: Installing InfiniBand and GPU drivers on a Rocky 10 compute node.
Action: Omnia detects that pre-compiled RHEL vendor drivers will not load on the Rocky kernel.
Result: Omnia seamlessly falls back to Dynamic Kernel Module Support (DKMS), compiling the CUDA and OFED drivers from source against the local Rocky kernel headers.

---

Scope Definition

Goals (Execution Outcomes)

• Support Rocky Linux 10.x as a selectable OS for base provisioning.
• Implement dynamic repository mapping to replace RHEL CDN requirements (AppStream, EPEL, CRB equivalents).
• Automate driver fallback to DKMS compilation for proprietary hardware (NVIDIA, Mellanox) when pre-compiled RHEL modules fail kernel signature checks.
• Support "Mixed-Mode" deployments (e.g., RHEL management nodes + Rocky compute nodes).
• Handle Secure Boot differences via Rocky's native keys.

Non-Goals (Scope Boundaries)

• Support for other RHEL clones (AlmaLinux, Oracle Linux) is out of scope for this specific capability unless they work for free implicitly via the Rocky logic.
• We will not maintain a secondary, separate set of Ansible playbooks. The logic must use dynamic when: ansible_distribution == 'Rocky' conditionals within the existing RHEL playbooks.

---

Requirements

Requirement 1: Dynamic Repository Mapping [Req 1]

The system must dynamically configure the correct open-source package repositories for Rocky Linux without expecting a Red Hat Subscription Manager (RHSM) entitlement.

Relates to: Use Case 1

Scenario: Base OS Package Installation

• When Omnia executes the base OS playbook, the system bypasses the RHSM registration tasks and configures the native Rocky BaseOS, AppStream, and CRB repositories to satisfy software dependencies.

Requirement 2: Kernel Module Compilation (DKMS) [Req 2]

The system must utilize DKMS or source compilation for proprietary drivers if RHEL pre-compiled binaries are rejected.

Relates to: Use Case 3

Scenario: OFED/CUDA Installation on Rocky

• When the standard RPM installation detects a kernel signature mismatch, Omnia automatically installs the Rocky kernel-devel headers and compiles the drivers locally via DKMS.

Requirement 3: Mixed-Mode Capability [Req 3]

The system must support heterogenous OS distributions between management planes and compute planes within the same cluster.

Relates to: Use Case 2

Scenario: Cross-OS Slurm Execution

• When Omnia completes the cluster setup, the system munge keys and Slurm configurations match perfectly across the OS boundary, and a test job executes successfully.

---

Usability

The OS selection (RHEL vs. Rocky) must be easily configurable per functional group within the standard omnia_config.yml or OpenCHAMI inventory.

---

Security

• Rocky Linux 10 nodes must properly integrate with UEFI Secure Boot, utilizing the correct Rocky-signed shims.

---

Telemetry

DKMS compilation for proprietary drivers on Rocky Linux must be monitored to track build success rates, compilation duration, and failure patterns. Engineering should utilize log aggregation (rsyslog or vector) to collect DKMS build logs from /var/lib/dkms/<module>/<version>/build/make.log and forward them to the central observability stack.

• DKMS compilation will inherently add time to the Day 1 provisioning process. This is an accepted tradeoff, but documentation should note the expected deployment time increase.

---

Documentation

• Admin Guide: Document the supported Mixed-Mode architectures and warn users about the increased deployment time associated with DKMS driver compilation.

---

Dependencies

• OS Images: Rocky 10.x ISOs/images must be available and validated by the Omnia QA team.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
QA Matrix Explosion	Operational	High	High	Testing every feature on both RHEL and Rocky effectively doubles the QA workload. Mitigate by heavily automating the CI/CD pipeline to run tests in parallel across both OS distributions.

---

Acceptance Criteria

• AC-001: Omnia successfully provisions a pure Rocky 10.x cluster, properly mapping AppStream, CRB, and EPEL repositories without RHSM errors. [Req 1]
• AC-002: Omnia successfully provisions a mixed-mode cluster (RHEL OIM/Controllers + Rocky Compute) and executes a multi-node MPI test. [Req 3]
• AC-003: Proprietary drivers (NVIDIA CUDA, Mellanox OFED) successfully compile and load on the Rocky 10 kernel via DKMS. [Req 2]
• AC-004: Rocky 10 nodes successfully boot with UEFI Secure Boot enabled. [Security, Req 1]
• AC-005: Implementation utilizes dynamic conditionals within existing RHEL playbooks rather than duplicating the entire codebase. [Scope, Req 1]

---

Assumptions / Open Questions

Assumptions

• A1: Rocky 10 remains 1:1 binary compatible with RHEL 10, meaning user-space applications (like Slurm and Kubernetes) will execute identically.

Open Questions

• None.

---