Slurm Prolog, Epilog, and Node Health Checks

This is a design-level spec, so start by agreeing scope with the maintainers. The payload names no existing file: work runs through the Omnia Ansible playbooks that render slurm.conf (Prolog, Epilog, HealthCheckProgram) and a new bundled sample-script directory. Done means acceptance criteria AC-001 to AC-005 pass on a test cluster.

Slurm Prolog, Epilog, and Node Health Checks

---

Summary

Cluster Administrators and Researchers need automated pre-flight validation, post-flight cleanup, and proactive health monitoring on compute nodes to prevent jobs from executing on degraded hardware. This capability utilizes native Slurm Prolog, Epilog, and HealthCheckProgram interfaces to verify GPU readiness, clear residual scratch data, and proactively drain unhealthy nodes, thereby drastically reducing job failure rates and manual recovery toil.

---

Problem Statement

In massive HPC environments, compute node health naturally fluctuates due to hardware degradation, stalled background processes, or residual state left behind by previous jobs (e.g., full scratch disks or stuck GPUs). Currently, if Slurm dispatches a new workload to an unhealthy or "dirty" node, the job will crash, wasting valuable scheduler time and forcing users to manually resubmit. Without integrated, configurable Slurm Prolog and Epilog scripts (to validate readiness before a job and clean up after) and periodic node health checks (to quarantine degraded nodes before jobs are assigned), cluster administrators suffer from high job failure rates, excessive manual node recovery toil, and poor overall cluster utilization.

---

Value Proposition

Customer Value

End users (researchers and data scientists) experience a massive reduction in job failures caused by infrastructure misconfigurations. Because nodes are automatically verified for GPU readiness and clean filesystems before their jobs execute, their workloads run predictably. For administrators, the cluster self-heals by automatically draining or rebooting unhealthy nodes before jobs are assigned to them, ensuring maximum uptime and efficiency.

---

User Personas

• HPC Systems Engineer: [Operating at the Slurm Control Plane layer to configure workload partitions, queues, and performance tuning] Manages the day-to-day operations of the HPC cluster. Needs automated mechanisms to prevent jobs from landing on degraded nodes and to clean up node state between users.
• End User / Researcher: Submits workloads to Slurm. Needs consistent, predictable job execution without random failures caused by "dirty" compute nodes.

---

Use Cases

Use Case 1: Job Lifecycle Validation and Cleanup

Actor: HPC Systems Engineer
Context: A user submits a GPU-intensive workload.
Action: Slurm executes the configured Prolog script before the job starts, and the Epilog script after it finishes.
Result: The Prolog verifies the GPU is healthy and accessible, allowing the job to run. The Epilog wipes the /tmp scratch space and resets the GPU state, ensuring the node is clean for the next user's job.

Use Case 2: Proactive Node Quarantining

Actor: HPC Systems Engineer
Context: A compute node experiences a silent filesystem unmount or memory ECC error while idle.
Action: Slurm executes the HealthCheckProgram on the idle node.
Result: The script detects the missing mount, safely marks the node state as DRAIN in Slurm to prevent new jobs from being assigned to it, and alerts the admin.

---

Scope Definition

Goals (Execution Outcomes)

• Provide Omnia deployment configuration to enable, disable, and define paths for Slurm Prolog, Epilog, and HealthCheckProgram scripts.
• Provide a bundled library of sample scripts for common use cases (e.g., GPU readiness, filesystem mount checks, temp cleanup, ECC checks).
• Enable automated failure policies where failed prologs block job execution and failed health checks drain the node.

Non-Goals (Scope Boundaries)

• Complex policy engines for dynamic script injection at the time of user job submission (users cannot supply arbitrary prologs).
• Remediation logic beyond basic Slurm state changes (e.g., Omnia will not automatically re-provision or physically power-cycle dead hardware in response to a failed script).

---

Requirements

Requirement 1: Prolog and Epilog Configuration [Req 1]

The system must configure Slurm to execute defined Prolog scripts before job initiation and Epilog scripts after job termination.

Relates to: Use Case 1 - Job Lifecycle Validation and Cleanup

Scenario: Prolog Failure Blocking

• When the configured Prolog script executes and returns a non-zero exit code (failure), Slurm aborts the job start, returns a clear user-facing error, and applies the configured failure policy (e.g., requeue the job or drain the node).

Scenario: Epilog Cleanup

• When the Epilog script executes, residual job state (e.g., temp files, per-job scratch directories, GPU allocations) is successfully cleared from all configured local scratch locations before the node accepts the next job.

Requirement 2: Periodic Node Health Checks [Req 2]

The system must configure Slurm's native HealthCheckProgram to execute periodically on compute nodes to proactively detect hardware and state degradation.

Relates to: Use Case 2 - Proactive Node Quarantining

Scenario: Idle Node Degradation Detected

• When the HealthCheckProgram executes and detects a failure (e.g., missing NFS mount), the script instructs Slurm to transition the node state to DRAIN, preventing future job assignments.

Scenario: Load-Aware Execution

• When the health check interval triggers during an active workload, the execution is deferred or gracefully skipped to prevent interrupting the workload's performance.

Requirement 3: Bundled Script Library [Req 3]

The system must provide a pre-packaged library of idempotent, secure sample scripts that administrators can select during Omnia deployment.

Relates to: Use Case 1 and 2

Scenario: Script Selection

• When the administrator selects the bundled GPU Health Check and Temp Cleanup scripts, Omnia automatically deploys them to the compute nodes with correct permissions and configures Slurm to execute them.

---

Reliability

Failure Mode	System Behavior
Hardware Degradation	The HealthCheckProgram proactively places degraded nodes into a DRAIN state, protecting overall cluster scheduling reliability by ensuring jobs only land on healthy hardware.

Availability: N/A
Recovery: Node recovery requires manual administrative intervention once DRAINED.

---

Telemetry

The Slurm prolog, epilog, and health check scripts must expose metrics to track execution success rates, failure patterns, and node health trends. Engineering should utilize a Slurm Prometheus exporter (such as slurm_prometheus_exporter) or log aggregation for prolog/epilog script output to expose this data to the central observability stack.

---

Security

• Scripts must be deployed by Ansible with strict permissions (e.g., root:root 755) to prevent unprivileged users from modifying execution logic.

---

Performance

HealthCheckProgram execution must be load-aware. It must be deferred or skipped during high CPU/GPU utilization to protect user job performance from unwanted monitoring overhead.

---

APIs

API	Type	Purpose	Consumers
slurm.conf	Configuration API	Omnia utilizes the native Prolog, Epilog, and HealthCheckProgram configuration interfaces within Slurm to orchestrate these checks.	Omnia Ansible Playbooks

---

Documentation

• Admin Guide: Provide documentation on how administrators can plug in their own custom scripts instead of the bundled samples, including required Slurm exit codes.

---

Dependencies

• Slurm Infrastructure: Slurm must be successfully deployed with the controller and daemons running.
• Hardware Tools: If bundled scripts perform GPU or driver checks (e.g., nvidia-smi), those drivers must be pre-installed on the compute nodes.
• Local Scratch Storage (BR-40709 / BR-75858): If epilog scripts are configured to clean locally mounted scratch drives, those drives must be provisioned and mounted via the automated drive mounting capability (BR-40709) or the runtime disk mount utility (BR-75858).

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Prolog Timeouts	Technical	Medium	Medium	If a custom Prolog script hangs, it delays scheduling for the entire cluster. Mitigate by enforcing strict execution timeouts in the Slurm configuration (PrologTimeout).
Driver Incompatibility	Operational	Medium	High	Bundled scripts may fail if the customer uses unexpected GPU driver versions. Mitigate by gating scripts with internal version checks and failing gracefully.

---

Acceptance Criteria

• AC-001: Omnia deploys and configures Prolog scripts to automatically run before every job, and Epilog scripts after every job. [Req 1]
• AC-002: A simulated Prolog failure successfully blocks the job from starting and enforces the configured failure policy (e.g., draining the node). [Req 1]
• AC-003: The configured HealthCheckProgram successfully executes on idle nodes, and transitions the node to a DRAIN state if a simulated hardware failure is detected. [Req 2]
• AC-004: Execution logs for prologs, epilogs, and periodic checks successfully record pass/fail status and durations to the specified log path. [Req 1]
• AC-005: The bundled sample scripts (e.g., GPU readiness, temp cleanup) deploy correctly and execute successfully on stateless and GPU-enabled nodes. [Req 3]

---

Assumptions / Open Questions

Assumptions

• A1: Required health tools/drivers (e.g. nvidia-smi) are present on the nodes.

Open Questions

• None.

---