Omnia TUI Provisioning Dashboard

The payload names no source files, so start by finding where Omnia runs Ansible provisioning playbooks and how it reports node state. The issue requires a Python TUI (e.g. textual or rich), polling of OpenCHAMI SMD/BSS APIs, Ansible callback capture, and a --no-tui fallback. Done means acceptance criteria AC-001 to AC-007 pass. This is design work, so agree on the architecture with maintainers before writing code.

Omnia TUI Provisioning Dashboard

---

Summary

HPC and AI Cluster Administrators require real-time visibility into the end-to-end bare-metal and software provisioning lifecycle to effectively deploy and troubleshoot Omnia clusters. The Omnia TUI Provisioning Dashboard is a terminal-based interface that aggregates hardware state from OpenCHAMI APIs and software state from Ansible callbacks. This unified view provides an instant assessment of cluster readiness and drastically reduces mean-time-to-resolution (MTTR) by isolating failures in massive deployment logs.

---

Problem Statement

Currently, tracking the deployment status of a large-scale Omnia cluster (e.g., 1000+ nodes) requires parsing massive, scrolling Ansible output logs and cross-referencing them manually with hardware BMC states. If a node fails during an early Redfish discovery phase, a PXE boot timeout, or late-stage GPU driver compilation, the failure is buried in a wall of text.

This lack of structured observability leads to severe "log fatigue" for HPC and AI cluster administrators. They spend excessive time pinpointing which specific node failed and at what exact lifecycle stage, delaying cluster readiness and degrading the overall perception of Omnia's robustness as an enterprise-grade provisioning tool.

---

Value Proposition

Customer Value

Administrators are completely shielded from raw Ansible log fatigue during cluster deployment. They receive a clean, live-updating matrix in their SSH terminal showing exactly which nodes are healthy, which are currently executing a specific playbook task, and which have failed. This observable change in experience turns cluster deployment from a black-box waiting game into a transparent, predictable process.

---

User Personas

• HPC/AI Systems Engineer: The primary deployer of the Omnia cluster. They run the provisioning commands from the control plane over SSH and need instant feedback on deployment progress and node health.
• Support Engineer: Support staff assisting with a failed deployment. They rely on the TUI to quickly drill down into specific node failure logs without requesting massive tarballs of Ansible output.

---

Use Cases

Use Case 1: Monitoring a Large-Scale Deployment

Actor: HPC/AI Systems Engineer
Context: The administrator has just initiated an Omnia deployment for a 500-node AI cluster.
Action: The administrator observes the TUI dashboard tracking the nodes as they transition from OpenCHAMI bare-metal discovery to OS booting and finally to Ansible software configuration.
Result: The administrator has high confidence in the deployment progress without needing to scroll through logs.

Use Case 2: Isolating a Deployment Failure

Actor: Support Engineer
Context: Three compute nodes fail to compile the NVIDIA driver during the post-OS software stack installation.
Action: The Support Engineer sees the nodes marked as "Failed" in red on the TUI matrix. They select the failed nodes to view the specific localized error output.
Result: The root cause (e.g., missing kernel headers) is identified in seconds rather than minutes of grep-searching raw log files.

---

Scope Definition

Goals (Execution Outcomes)

• Goal 1: Provide a live, terminal-based dashboard that displays the status of all nodes in an Omnia deployment.
• Goal 2: Integrate with OpenCHAMI SMD and BSS APIs to report pre-OS hardware state.
• Goal 3: Integrate with Ansible callbacks to report post-OS software execution state.

Non-Goals (Scope Boundaries)

• Non-Goal 1: A graphical Web UI version of this dashboard (this remains strictly a terminal UI).
• Non-Goal 2: Long-term historical analytics or persistent logging storage beyond the active deployment session.
• Non-Goal 3: Day-2 continuous health monitoring or always-on daemon tracking for NOC operations. This is addressed by the complementary Grafana Telemetry Dashboards capability (BR-32936).

---

Requirements

Requirement: TUI Rendering Engine [Req 1]

The system must render a live-updating text-based user interface in the terminal during an Omnia deployment.

Scenario: Successful rendering

• When the terminal supports standard ANSI escape sequences, a structured UI matrix of nodes is rendered instead of standard STDOUT logging.

Scenario: Fallback to raw logs

• When the terminal does not support a TUI (e.g., a CI/CD pipeline or redirected output), the system automatically degrades to standard Ansible STDOUT logging.

Requirement: OpenCHAMI API Integration [Req 2]

The system must poll OpenCHAMI APIs to display hardware discovery and boot states.

Scenario: Hardware in discovery

• When a node is queried via the OpenCHAMI SMD API, the TUI displays the node state as "Hardware Discovery".

Scenario: API Unreachable

• When the OpenCHAMI API times out or returns a 500 error, the TUI displays the node state as "Unknown (API Error)".

Requirement: Ansible Callback Integration [Req 3]

The system must capture automation playbook progress to display software configuration states.

Scenario: Task execution

• When Ansible begins the "Install GPU Drivers" task, the TUI updates the node state to "Installing GPU Drivers".

Scenario: Task failure

• When an Ansible task returns a fatal error, the TUI highlights the node in red and records the failure reason.

---

Usability

• The TUI must support keyboard navigation (arrow keys, Enter, Esc) to drill down into failed nodes.
• The TUI must remain responsive and not lock up the terminal, even when monitoring 1000+ nodes.
• The dashboard matrix must utilize clear color coding (e.g., Green for Up, Yellow for Provisioning, Red for Failed).
• Error messages must be easily copy-pasteable by the user.

---

Reliability

Failure Mode	System Behavior
OpenCHAMI API timeout	TUI marks node status as "Stale/Unknown" and retries with backoff.
SSH connection drop to node	TUI reflects the Ansible unreachable error gracefully without crashing.
Terminal resize event	TUI dynamically redraws the matrix to fit the new window dimensions.

Availability: N/A - Runs as a transient process during deployment.
Recovery: If the TUI process crashes, the deployment playbook must continue running in the background.

---

Security

• API polling to OpenCHAMI must utilize secure, authenticated endpoints (e.g., via OPAAL tokens).
• The TUI must not display sensitive secrets or passwords on the screen if they are printed in Ansible logs.
• The TUI process must execute with the minimum privileges required to run Ansible and query OpenCHAMI.

---

Performance

Metric	Target
Dashboard rendering latency	< 100ms per frame
OpenCHAMI API polling impact	Minimal load (e.g., bulk queries rather than per-node loops)
Max concurrent nodes monitored	1,000+ nodes without terminal lag

---

Telemetry

The TUI dashboard must expose metrics to track rendering performance, API polling success rates, and user interaction patterns. Engineering should utilize OpenTelemetry instrumentation for terminal applications to expose this data to the central observability stack.

---

Documentation

• User Guide Updates: The Omnia deployment documentation must be updated with screenshots and instructions on how to use the TUI.
• Release Notes: The feature must be highlighted in the major release announcement.

---

Dependencies

• OpenCHAMI Core: Relies on the SMD and BSS microservices being deployed and stable (v1.x or higher).
• Ansible Core: Relies on Ansible >= 2.15 to support advanced callback plugins.
• Python Environment: The control plane must have the required TUI rendering libraries (e.g., textual or rich) installed.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
OpenCHAMI API Rate Limiting	Technical	High	Medium	Implement bulk queries instead of 1:1 polling loops.
TUI Rendering Performance at Scale	Technical	High	Low	Use optimized text rendering libraries and pagination for large node counts.
User Resistance to TUI	Adoption	Low	Low	Provide a CLI flag (--no-tui) to fall back to traditional Ansible output.

---

Acceptance Criteria

• AC-001: The TUI successfully intercepts an Omnia deployment and renders a visual node matrix in the terminal. [Req 1]
• AC-002: The TUI automatically falls back to raw Ansible output if executed in a non-interactive shell. [Req 1]
• AC-003: The dashboard displays accurate hardware states (e.g., "Hardware Discovery") by polling the OpenCHAMI SMD API. [Req 2]
• AC-004: The dashboard displays accurate software execution states (e.g., "Installing GPU Drivers") by capturing automation playbook events. [Req 3]
• AC-005: Nodes that experience a fatal Ansible error are marked in red and the specific error message is accessible via keyboard navigation. [Req 3]
• AC-006: The TUI maintains < 100ms render latency when monitoring a simulated 1,000+ node cluster. [Req 1]
• AC-007: Polling the OpenCHAMI API does not generate HTTP 429 Too Many Requests errors during standard operation. [Req 2]

---

Assumptions / Open Questions

Assumptions

• A1: Required underlying hardware and OS capabilities are present.

Open Questions

• None.

---