UFM Agent for hosts using Infiniband

The payload names no source files. It refers to the standard Omnia inventory.yml and to inventory groups such as [infiniband_supernics] and [infiniband_dpus], so start by checking how existing inventory groups drive the provisioning playbooks. The issue is a design-level feature with open scope and a required pre-install compatibility check, so a newcomer should first discuss the approach with maintainers. Done means an IB-tagged node installs the agent, starts its service, and passes the ibstat and ib_write_bw smoke test.

UFM Agent for hosts using Infiniband

---

Summary

The NVIDIA Unified Fabric Manager (UFM) platform requires a UFM Host Agent installed on compute nodes to provide the UFM appliance with deep host-level visibility, telemetry, and remote management capabilities (such as OFED and HCA firmware upgrades). Currently, installing and configuring this agent on InfiniBand-equipped hosts—specifically those utilizing NVIDIA BlueField-3 DPUs and SuperNICs—is a manual, error-prone process. This capability automates the detection, installation, and configuration of the UFM Host Agent during Omnia's Day 1 cluster provisioning, ensuring 100% compliance with UFM prerequisites at scale.

---

Problem Statement

While the NVIDIA UFM appliance controls the InfiniBand core switches and BlueField-3 network interface cards (HCAs), it relies on an embedded UFM Host Agent on the compute nodes to facilitate complete end-to-end communication. The agent is responsible for reporting IP/CPU/Memory data, streaming host performance telemetry, managing IP interfaces over InfiniBand partitions, and enabling remote firmware/OFED updates.

In massive scale-out deployments, manually downloading the agent tarballs, installing prerequisite dependencies, injecting license files, and applying specific configurations (differentiating between north-south DPUs and east-west SuperNICs) introduces massive operational overhead. This manual configuration bottleneck slows down cluster deployment by days and frequently results in misconfigured hosts that the UFM Server cannot discover or manage.

---

Value Proposition

Customer Value

Cluster Administrators achieve a true "plug-and-play" InfiniBand management experience. By simply defining their InfiniBand hardware in the Omnia inventory, the compute nodes are automatically provisioned with the correct UFM Host Agent and configuration, making them immediately visible and manageable within the central UFM Appliance GUI upon boot.

---

User Personas

• HPC Systems Engineer: [Operating at the Slurm Control Plane layer to configure workload partitions, queues, and performance tuning] Requires all InfiniBand-equipped compute nodes to appear cleanly in the UFM appliance without executing manual post-deployment scripts.
• Network Administrator: [Managing IB fabrics, BGP routing, and switch-level enforcement outside the host OS] Relies on the host telemetry and remote management capabilities exposed by the UFM Host Agent to maintain the health of the InfiniBand fabric.

---

Use Cases

Use Case 1: Automated Agent Deployment

Actor: System (Omnia Provisioning)
Context: A new compute node equipped with a BlueField-3 SuperNIC is provisioned.
Action: Omnia detects the presence of the InfiniBand hardware, downloads the UFM Agent package from the local Omnia repository, installs the required prerequisites, and configures the agent service.
Result: Upon node boot, the UFM Host Agent starts automatically, connects to the central UFM Server, and the host immediately appears in the UFM Device Inventory.

Use Case 2: DPU vs SuperNIC Configuration

Actor: System (Omnia Provisioning)
Context: A cluster contains a mix of nodes with DPUs (for north-south storage traffic) and SuperNICs (for east-west compute traffic).
Action: Omnia reads the inventory definitions and applies the specific UFM Host Agent configuration profile required for the respective hardware type.
Result: The UFM server correctly categorizes and manages the interfaces based on their designated traffic patterns.

---

Scope Definition

Goals (Execution Outcomes)

• Omnia will download the NVIDIA UFM Host Agent installation packages (TGZ/RPM) to the local Omnia repository.
• Detect the presence of InfiniBand hardware (or utilize a targeted Ansible inventory group) to selectively apply the agent installation.
• Install the UFM Host Agent and all necessary OS prerequisites during node provisioning (e.g., via Ansible tasks or cloud-init).
• Apply the appropriate initial configuration files, distinguishing between DPU and SuperNIC architectures.
• Ensure the UFM Host Agent service is enabled and started upon node boot.
• Provide a Day 1 validation playbook for host-side smoke testing to confirm InfiniBand port status and end-to-end RDMA connectivity.

Non-Goals (Scope Boundaries)

• Omnia will not automatically deploy or configure the central UFM Server/Appliance itself; the UFM Server is assumed to be an existing customer prerequisite.
• Omnia will not attempt to actively manage the UFM licenses; the customer must provide the required UFM licensing, though Omnia may assist in placing the license file on the host if required by the agent.

---

Requirements

Requirement 1: Hardware Detection and Installation [Req 1]

The system must detect nodes designated for InfiniBand networking and automatically install the UFM Host Agent and its prerequisites during provisioning.

Relates to: Use Case 1

Scenario: Base Installation

• When Omnia executes the provisioning playbooks, the system installs the UFM Host Agent dependencies, deploys the agent binaries from the local repo, and enables the systemd service.

Requirement 2: Hardware-Specific Configuration [Req 2]

The system must apply distinct agent configuration profiles based on whether the host is equipped with a DPU (e.g., B3220) or a SuperNIC (e.g., B3140H).

Relates to: Use Case 2

Scenario: DPU Configuration

• When the UFM Host Agent is configured, Omnia injects the DPU-specific configuration file, ensuring the UFM server recognizes it for north-south traffic management.

Requirement 3: Day 1 Fabric Health Validation (Smoke Test) [Req 3]

The system must include a Day 1 validation playbook to verify the InfiniBand fabric at the host level, executing basic smoke tests prior to cluster handoff.

Scenario: RDMA Smoke Test

• When the validation playbook is executed, the system runs ibstat to confirm ports are ACTIVE and executes ib_write_bw (or perftest) between the node pair to validate end-to-end RDMA connectivity.

---

Usability

Administrators will control the deployment by assigning nodes to specific inventory groups (e.g., [infiniband_supernics], [infiniband_dpus]) within the standard Omnia inventory.yml.

---

Reliability

Failure Mode	System Behavior
UFM Server Unreachable	If the host agent cannot reach the UFM Server at boot, it will retry connection via systemd without blocking the OS boot process or Slurm daemon startup. It must emit an explicit CRITICAL log event to Vector on consecutive failures; it must not fail silently.

---

Security

• Installation binaries must be served securely from the trusted local Omnia repository rather than fetching directly from the internet during deployment.

---

Performance

• The installation of the agent via Ansible or cloud-init must NOT add more than 2 minutes to the overall node provisioning time.

---

Documentation

• Admin Guide: Document the required inventory group definitions to trigger the agent installation. Provide instructions on how customers should populate the local Omnia repository with the UFM Agent binaries prior to deployment.

---

Dependencies

• Software Binaries: The site team must supply the official NVIDIA UFM Host Agent TGZ/RPM files to the Omnia control plane.
• UFM Appliance: A central NVIDIA UFM Server must be active and reachable over the management network for the agents to register.
• Hardware: Compatible NVIDIA BlueField-3 DPUs or SuperNICs installed in the target nodes.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
OS Incompatibility	Technical	High	Medium	NVIDIA frequently ties agent versions to specific OS kernels/OFED versions. Mitigate by enforcing strict version compatibility checks in the automation playbook before attempting installation.

---

Acceptance Criteria

• AC-001: An administrator can execute the standard Omnia deployment and successfully install the UFM Host Agent on target nodes. [Req 1]
• AC-002: Omnia correctly deploys distinct configurations for DPUs versus SuperNICs based on the node's inventory grouping. [Req 2]
• AC-003: Managed servers with DPUs/SuperNICs successfully boot, start the agent service, and become visible in the central UFM Appliance GUI. [Req 1]
• AC-004: The Omnia provisioning script validates that necessary system prerequisites (e.g., OFED dependencies) are present before installing the agent. [Req 1]
• AC-005: The deployment logs clearly capture the success, failure, or intentional skipping of the UFM Host Agent installation. [Supportability, Req 1]
• AC-006: A Day 1 host-side validation playbook exists and successfully demonstrates ibstat port verification and RDMA bandwidth (ib_write_bw/perftest) between sample compute nodes. [Req 3]

---

Scope & Sequencing

Targeted for initial cluster deployment. Must be sequenced alongside base OFED driver installation.

---

Assumptions / Open Questions

Assumptions

• A1: Customers will download the proprietary UFM Host Agent binaries from the NVIDIA Licensing Portal and place them in the Omnia artifact directory.

Open Questions

• None.

---