RedHat MPAM enablement in RHEL 10.x

Start with the enable_arm_mpam toggle in omnia_config.yml and the cgroup.conf template named in Requirement 3, then confirm whether the boot-parameter mechanism from BR-36094 exists in the repo yet. Requirement 3 and the GRUB injection depend on that mechanism and on Red Hat kernel support that is not yet available. Done means AC-001 to AC-004 hold: MPAM parameters are injected on capable ARM nodes, /sys/fs/resctrl is mounted, x86 nodes are skipped with an informational message, and cgroup.conf is templated.

RedHat MPAM enablement in RHEL 10.x

---

Summary

ARM Memory Partitioning and Monitoring (MPAM) provides hardware-enforced isolation of L3 cache and memory bandwidth, acting as ARM's equivalent to Intel RDT. While Red Hat plans to natively support MPAM in the RHEL 10.x kernel (and potentially in specialized builds like Red Hat Enterprise Linux for NVIDIA / RHEL AI), the capability is disabled by default. This specification ensures Omnia explicitly enables MPAM via GRUB bootloader parameters during Day-1 provisioning on supported ARM hardware (e.g., NVIDIA Grace), and configures the resctrl filesystem so that Slurm and Kubernetes can utilize it for dynamic workload isolation.

---

Problem Statement

Modern HPC environments are increasingly moving towards "co-scheduled" architectures, where traditional Slurm batch jobs and modern Kubernetes services run simultaneously on the same bare-metal compute nodes. However, when these workloads share the same memory subsystem on ARM-based superchips, a "noisy neighbor" effect occurs. A rogue Kubernetes pod can thrash the L3 cache or monopolize the memory bandwidth, starving a critical physics simulation or AI training job running in Slurm, leading to unpredictable execution times.

Although ARM MPAM hardware solves this by enforcing strict limits on cache and bandwidth utilization, the OS plumbing to activate it is complex. If Omnia does not automatically inject the correct kernel parameters and mount the resctrl interfaces during node provisioning, administrators are left with hardware they cannot fully utilize, violating the performance guarantees required by major customers at this scale.

---

Value Proposition

Customer Value

Customers can safely maximize their hardware utilization by co-scheduling disparate workloads (Slurm and K8s) on the same ARM nodes without fear of performance degradation. MPAM guarantees that high-priority workloads receive their allocated L3 cache and memory bandwidth, resulting in predictable "time-to-science."

---

User Personas

• HPC Systems Engineer: Needs to ensure that physical hardware boundaries can be enforced at the OS level so they can confidently pack more workloads onto expensive ARM compute nodes.
• Researchers/End-Users: Require predictable application performance even when other tenants or background cluster services are heavily loading the memory subsystem.

---

Use Cases

Use Case 1: Intra-Slurm Isolation

Actor: Slurm Workload Manager (Automated)
Context: Two Slurm jobs are scheduled on the same ARM compute node.
Action: Omnia has pre-configured Slurm's cgroup.conf and task/cgroup plugins to be resctrl-aware.
Result: Slurm automatically allocates isolated L3 cache partitions and memory bandwidth limits to Job A and Job B independently, preventing either from starving the other.

Use Case 2: Co-scheduled Kubernetes and Slurm

Actor: System (Automated)
Context: A node runs both Slurm compute workers and Kubernetes workload pods.
Action: Omnia enables MPAM via the GRUB bootloader and configures both the Kubelet and Slurm daemon to utilize the resctrl interfaces.
Result: Administrators can set global policies guaranteeing that Slurm jobs receive 70% of the memory bandwidth, strictly capping Kubernetes pods at 30%.

---

Scope Definition

Goals (Execution Outcomes)

• Dynamically detect ARM architecture and MPAM hardware support during OS provisioning.
• Inject the required kernel parameters (e.g., mpam=on) into the GRUB bootloader configuration for RHEL 10.x.
• Ensure the resctrl (Resource Control) filesystem is mounted and accessible on compute nodes.
• Configure Slurm and Kubernetes resource management plugins to hook into resctrl if co-scheduling is detected.

Non-Goals (Scope Boundaries)

• Omnia will NOT write, compile, or provide custom MPAM kernel modules. This capability strictly relies on Red Hat natively supporting the feature upstream in RHEL 10.x (or specialized NVIDIA-specific RHEL builds).
• Omnia will NOT dynamically create the partitions on the fly when a job is submitted. Omnia simply builds the OS-level plumbing so Slurm and K8s can do the dynamic slicing.

---

Requirements

Requirement 1: MPAM Boot Parameter Injection [Req 1]

The system must inject MPAM enablement parameters (e.g., mpam=on) into the bootloader of supported ARM nodes using the declarative kernel parameter management infrastructure provided by BR-36094.

Relates to: Use Case 1 and 2

Scenario: RHEL 10 Provisioning on NVIDIA Grace

• When the hardware detection playbook executes, Omnia identifies MPAM capability and leverages BR-36094's boot parameter injection mechanism to append the necessary flags to the GRUB command line (or BSS payload for PXE boot), ensuring the MPAM controller initializes at boot.

Requirement 2: Resctrl Filesystem Mounting [Req 2]

The system must ensure the OS exposes the hardware partitions to the user-space workload managers.

Relates to: Use Case 1

Scenario: Filesystem Verification

• When Omnia executes the post-boot validation, the /sys/fs/resctrl directory is successfully mounted and accessible to the Slurm daemon and Kubelet services.

Requirement 3: Workload Manager Configuration [Req 3]

The system must configure Slurm and K8s to utilize the hardware partitions if they are deployed together.

Relates to: Use Case 2

Scenario: Co-scheduling Plumbing

• When the configuration templates are applied, the system ensures Slurm's cgroup.conf is populated with the correct resctrl parameters, enabling hardware-enforced isolation.

---

Usability

The enablement of MPAM should be a toggleable boolean in omnia_config.yml (e.g., enable_arm_mpam: true), allowing customers to disable it if they prefer standard software-level cgroups.

---

Reliability

Failure Mode	System Behavior
Hardware Mismatch	If enable_arm_mpam is true but the physical CPU does not support MPAM (e.g., an x86 node), Omnia must gracefully skip the GRUB injection and log an informational message rather than failing the provisioning run.

---

Performance

• Enabling MPAM introduces negligible OS overhead but dramatically increases deterministic performance for high-priority workloads.

---

Telemetry

ARM MPAM enablement and resctrl filesystem configuration must be monitored to track successful hardware enablement and partition effectiveness. Engineering should utilize the Linux resctrl filesystem interface at /sys/fs/resctrl with node_exporter or a custom collector to read monitoring data from sysfs and expose it to the central observability stack.

---

Documentation

• Admin Guide: Document the specific versions of RHEL 10 (e.g., Red Hat Enterprise Linux AI / NVIDIA builds) that support MPAM, and provide guidance on how administrators can verify their L3 cache partition metrics.

---

Dependencies

• BR-36094: Automated Kernel Boot Parameter Management: Provides the foundational architecture for declarative kernel parameter injection (e.g., mpam=on) into both OpenCHAMI BSS (PXE boot) and GRUB2 (diskful boot) without manual edits. This spec leverages BR-36094 to inject the required MPAM enablement parameters.
• BR-22025: RHEL for NVIDIA Support for GPU Nodes: Enables RHEL for NVIDIA as a supported OS variant for aarch64 functional groups. MPAM enablement is scoped to ARM nodes running RHEL for NVIDIA or specialized RHEL AI builds, which require this OS variant support.
• JA-17289: Workload Kubernetes Cluster: Required for true co-scheduling use cases where Kubernetes jobs are placed alongside Slurm jobs on the same worker nodes.
• Red Hat Upstream Support: Red Hat must release a RHEL 10.x kernel (or RHEL AI / Red Hat Enterprise Linux for NVIDIA build) that natively includes the ARM MPAM drivers.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Upstream Delays	Technical	High	High	Red Hat may delay the inclusion of MPAM into the mainline RHEL 10 kernel. Mitigate by closely tracking Red Hat's roadmap for "Red Hat Enterprise Linux for NVIDIA 26.01" , and keeping the Omnia capability marked as experimental until upstream support stabilizes.

---

Acceptance Criteria

• AC-001: Omnia successfully detects ARM MPAM hardware during provisioning and injects the required GRUB boot parameters. [Req 1]
• AC-002: The /sys/fs/resctrl filesystem is verified to be mounted and active on supported nodes post-boot. [Req 2]
• AC-003: Omnia gracefully bypasses MPAM configuration on x86 nodes or ARM nodes lacking MPAM support without throwing an error. [Reliability, Req 1]
• AC-004: Slurm's cgroup.conf is successfully templated with the correct resctrl parameters. [Req 3]

---

Scope & Sequencing

Must be sequenced after BR-36094 (kernel boot parameter management infrastructure) and BR-22025 (RHEL for NVIDIA OS variant for ARM), and before the final configuration of the Slurm and Kubernetes daemons.

---

Assumptions / Open Questions

Assumptions

• A1: Red Hat will include the necessary resctrl interfaces for MPAM in their standard RHEL 10.x release or specialized "Red Hat Enterprise Linux for NVIDIA" builds.

Open Questions

• None.

---