Slurm QoS Priority-Triggered Network and Node Configuration

The work centres on a Slurm Prolog script that reads a YAML priority mapping from Omnia inventory files and sets InfiniBand service levels and virtual lanes on the compute node. Start by reading the Prolog/Epilog framework that JA-17448 is meant to provide, since this issue depends on it, and confirm whether that framework exists yet. Done means the acceptance criteria AC-001 to AC-007 hold: valid mappings apply without reboot, invalid ones fail with audit logging, and rollback restores the prior configuration.

Slurm QoS Priority-Triggered Network and Node Configuration

---

Summary

HPC users rely on Slurm Quality of Service (QoS) levels to assign priorities to their workloads (e.g., low, medium, mission-critical). However, without hardware enforcement, a high-priority job can still suffer from network congestion caused by low-priority background jobs on the same InfiniBand fabric. This capability integrates Slurm QoS with the physical network by dynamically configuring InfiniBand Virtual Lanes (VL) and Service Levels (SL) on compute nodes. Utilizing the Slurm Prolog/Epilog framework, Omnia automatically tunes the node's network parameters to match the job's priority—without requiring a reboot—ensuring true end-to-end performance guarantees.

---

Problem Statement

In dense multi-tenant HPC clusters, network congestion on the InfiniBand fabric is a major bottleneck. Slurm administrators use multifactor priority and QoS definitions to sequence when a job starts, but once jobs are running, they all compete equally for physical network bandwidth. A low-priority batch job executing heavy I/O can saturate the fabric, starving a highly sensitive, mission-critical MPI workload of the bandwidth and latency it needs.

Currently, administrators lack an automated mechanism to translate a Slurm job's logical priority into hardware-level network enforcement. Modifying InfiniBand Virtual Lanes and Service Levels requires manual configuration, which is impossible to do dynamically per-job at scale. This gap prevents clusters from offering true SLA-backed performance tiers, leading to unpredictable job completion times for critical research and enterprise workloads.

---

Value Proposition

Customer Value

Cluster Administrators can offer guaranteed network SLAs. A researcher running a mission-critical workload can select the "High" Slurm QoS and trust that their job's traffic will be isolated into a high-priority InfiniBand Virtual Lane, completely bypassing congestion caused by "Low" priority jobs. The dynamic configuration requires zero node reboots, maximizing cluster uptime.

---

User Personas

• HPC Systems Engineer: [Operating at the Slurm Control Plane layer to configure workload partitions, queues, and performance tuning] Defines the mapping between Slurm QoS levels and InfiniBand Service Levels. Needs audit logs and safe rollbacks if configurations fail.
• End User / Researcher: Submits workloads using Slurm QoS flags. Expects their high-priority jobs to execute with dedicated, unhindered network performance.

---

Use Cases

Use Case 1: Dynamic Network Reconfiguration for High-Priority Jobs

Actor: System (Automated via Prolog)
Context: A user submits a job requesting the "mission-critical" Slurm QoS.
Action: As the job starts, Omnia's Slurm Prolog script reads the priority mapping configuration and dynamically reconfigures the compute node's InfiniBand interface to use the designated high-priority Service Level.
Result: The job's network traffic is routed through a dedicated Virtual Lane on the IB switches, avoiding congestion. The node does not reboot.

Use Case 2: Graceful Failure and Rollback

Actor: System (Automated)
Context: Omnia attempts to apply an invalid priority network configuration or encounters a driver error during the Prolog execution.
Action: The system detects the failure, aborts the application, and triggers a rollback.
Result: The compute node is restored to its previous working network configuration without network disruption, and an audit log is generated detailing the failure.

Use Case 3: Priority Mapping Updates

Actor: HPC Systems Engineer
Context: The administrator decides to add a new "Ultra-High" QoS tier.
Action: The admin updates the priority mapping configuration file in Omnia.
Result: Omnia automatically detects the file update and applies the new Service Level mappings to the Slurm nodes during the next configuration sync or job execution.

---

Scope Definition

Goals (Execution Outcomes)

• Define a configuration standard (mapping file) that links Slurm QoS levels to InfiniBand Service Levels (SL) and Virtual Lanes (VL).
• Utilize the Slurm Prolog/Epilog hook framework to dynamically apply and revert these settings on compute nodes at job start/end.
• Ensure configurations are applied "hot" (without requiring a full node reboot).
• Implement robust error handling: log configuration changes, fail gracefully on bad inputs, and automatically rollback on failure.

Non-Goals (Scope Boundaries)

• Reconfiguring the core InfiniBand switches themselves. This capability assumes the core switch fabric is already configured by the customer/UFM to respect and route the distinct Virtual Lanes. This capability only configures the compute node's egress tagging.

---

Requirements

Requirement 1: Priority Mapping Execution [Req 1]

The system must read a priority mapping configuration file and correctly configure the compute node's InfiniBand Service Levels and Virtual Lanes according to the job's Slurm QoS.

Relates to: Use Case 1

Scenario: Valid Mapping Application

• When a Slurm job with a designated QoS starts, Omnia configures the node to the corresponding InfiniBand service level without requiring a reboot.

Scenario: Invalid Configuration

• When Omnia attempts to apply it, the system fails gracefully with clear error messaging in the audit log and causes zero network disruption to the node.

Requirement 2: Resilience and Auditing [Req 2]

The system must audit all QoS-triggered network changes and support automated rollback to a known-good state if the application fails.

Relates to: Use Case 2

Scenario: Network Configuration Rollback

• When Omnia detects the error, the system automatically restores the previous working configuration to prevent stranding the node offline.

Scenario: Change Auditing

• When the configuration is applied, an audit log entry is created detailing the requested QoS, the applied network parameters, and the success/failure status.

---

Usability

The priority mapping configuration will be defined via standard Omnia inventory files (e.g., YAML), keeping the administrative experience consistent with other networking configurations.

---

Reliability

Failure Mode	System Behavior
Prolog Script Timeout	If the network reconfiguration hangs, Slurm will time out the Prolog (as per JA-17448 policies), fail the job safely, and optionally drain the node. The rollback logic will attempt to restore default networking.

---

Telemetry

The Slurm QoS network configuration must expose metrics to track network reconfiguration success rates, rollback frequency, and performance impact on job scheduling. Engineering should utilize a Slurm Prometheus exporter (such as slurm_prometheus_exporter or the native OpenMetrics support in Slurm 25.11+) to expose this data to the central observability stack.

---

Performance

• The network reconfiguration must execute rapidly within the Slurm Prolog window (typically < 5 seconds) to avoid delaying job startup times.

---

Documentation

• HPC Admin Guide: Detail the syntax for the priority mapping configuration file. Explicitly state the prerequisite that the customer's InfiniBand switch fabric must be pre-configured to respect the specified Service Levels and Virtual Lanes.

---

Dependencies

• JA-17448: Slurm Prolog, Epilog, and Node Health Checks: This is a critical prerequisite. The dynamic application of the network configuration is fundamentally an advanced application of the Prolog/Epilog execution framework.
• BR-81934 / JA-8825: Networking Baselines: The underlying IPoIB and bonded network configurations must be stable before dynamically manipulating their traffic lanes.
• Physical Fabric: The InfiniBand Subnet Manager (e.g., UFM or OpenSM) and physical switches must be configured by the customer to route and prioritize the targeted Service Levels.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Switch Misalignment	Operational	Medium	High	If Omnia tags traffic for a high-priority VL but the switch drops it due to lack of configuration, the job will fail. Mitigate through explicit prerequisite documentation.

---

Acceptance Criteria

• AC-001: When Omnia processes a valid priority mapping configuration file, corresponding InfiniBand service levels and virtual lanes are configured correctly. [Req 1]
• AC-002: When Omnia attempts to apply an invalid priority mapping configuration, the system fails gracefully with clear error messaging and no network disruption. [Req 1]
• AC-003: When a high-priority job's corresponding network configuration is applied, the node achieves the configured service level without requiring a full reboot. [Req 1]
• AC-004: When Omnia applies configurations with multiple priority levels, each priority level receives distinct network settings as specified. [Req 1]
• AC-005: When Omnia applies a configuration change, an audit log entry is created documenting the change and its success/failure status. [Req 2]
• AC-006: When a network configuration failure occurs and Omnia attempts rollback, the previous working configuration is restored successfully. [Req 2]
• AC-007: When a Slurm admin modifies the priority mapping file, Omnia detects and applies the changes on the next configuration run. [Req 1]

---

Scope & Sequencing

• Must sequence AFTER JA-17448 (Slurm Prolog/Epilog Framework). Engineering cannot build the QoS network trigger until the pre/post job execution hook framework exists.
• Must sequence after core IB networking baselines (BR-81934, JA-8825).

---

Assumptions / Open Questions

Assumptions

• A1: Slurm provides the job's QoS level as an accessible environment variable during the Prolog execution phase.

Open Questions

• None.

---