Networking: Bonded Interfaces in active-backup mode (Ethernet & InfiniBand)

The issue names no files, tests or entry points, so first find where the Omnia inventory defines network interfaces and how Day 1 provisioning applies them. Its requirements are to accept an active-backup (mode=1) bond over two physical ports, add 802.1Q VLANs or PKeys on top, and fail explicitly on a missing slave. Done means a tested inventory schema and playbook path for those cases, which needs design agreement first.

Networking: Bonded Interfaces in active-backup mode (Ethernet & InfiniBand)

---

Summary

In production HPC environments, compute nodes require highly available connections to shared storage clusters over InfiniBand (IB) or Ethernet networks. This capability enhances Omnia's Day 1 provisioning to support network bonding in active-backup mode (Mode 1). By aggregating two physical network ports (either on dual single-port NICs or a single double-port NIC) into a single logical bonded interface, network traffic automatically fails over to the standby link if the primary link fails, ensuring zero disruption to storage mounts and running Slurm jobs.

---

Problem Statement

Currently, Omnia network provisioning only configures single, discrete network interfaces on compute nodes. In rigorous production HPC environments, compute nodes maintain constant, high-throughput connections to shared storage clusters (e.g., NFS, PowerScale). Without network redundancy, a single point of failure—such as a bad switch port, severed cable, or failed NIC—instantly severs the compute node's connection to the storage fabric. This results in dropped storage mounts, crashed Slurm jobs, corrupted in-flight data, and severely degraded cluster reliability. Administrators are forced to manually configure Link Aggregation Groups (LAGs/bonding) on every node post-deployment, negating the value of automated cluster provisioning.

---

Value Proposition

Customer Value

Cluster Administrators can declaratively configure highly available network paths for their compute nodes directly through the Omnia inventory. By utilizing active-backup bonding, they ensure seamless link failover, protecting their multi-day computational jobs and massive storage transfers from physical network disruptions.

---

User Personas

• Infra/Network Administrator: Responsible for designing the physical network topology and translating it into the Omnia configuration inventory to ensure fault tolerance.
• HPC Systems Engineer: [Operating at the Slurm Control Plane layer to configure workload partitions, queues, and performance tuning] Relies on the underlying network redundancy to ensure Slurm jobs do not fail due to transient link drops.

---

Use Cases

Use Case 1: Compute Node Storage Redundancy (Dual or Single NIC)

Actor: Infra Admin
Context: Provisioning compute nodes connected to a highly available storage cluster over InfiniBand or Ethernet. The node might have two separate single-port NICs or one double-port NIC.
Action: The administrator defines a bonded interface (bond0) operating in active-backup mode across two physical ports (e.g., ib0, ib1 or eth1, eth2) in the Omnia inventory.
Result: Omnia provisions the OS networking stack (e.g., NetworkManager or a similar tool) to aggregate the interfaces. If the active link/port fails, the backup link instantly assumes the MAC/GUID address and traffic, preserving the storage mount. This provides cable/switch redundancy (in both single and dual NIC setups) and full card redundancy (only in dual NIC setups).

Use Case 2: Tagged VLANs on Bonded Interfaces

Actor: Infra Admin
Context: The storage and telemetry networks are segregated into different VLANs but share the same physical redundant uplink.
Action: The administrator defines multiple VLAN interfaces (e.g., bond0.100, bond0.200) on top of the newly defined bond0 interface.
Result: Omnia correctly provisions the bonded interface and the subsequent 802.1Q VLAN tags, allowing highly available, segmented traffic across the dual links.

---

Scope Definition

Goals (Execution Outcomes)

• Support declarative configuration of Linux bonded interfaces in the Omnia network inventory for both Ethernet and InfiniBand (IB) interfaces.
• Specifically support active-backup bonding (Mode 1), which does not require complex switch-side LACP negotiation and is the standard for InfiniBand redundancy.
• Support configuring single or multiple bonded interfaces per node across any physical port topology (single double-port NICs or dual single-port NICs).
• Support deploying tagged VLANs (Ethernet) or PKeys (InfiniBand) directly on top of logical bonded interfaces.
• Apply configurations persistently during Day 1 node provisioning.

Non-Goals (Scope Boundaries)

• Dynamic Link Aggregation (LACP / Mode 4) is out of scope for this requirement as per leadership guidance; the focus is strictly on failover (active-backup), not throughput aggregation.
• Switch-side configuration (e.g., configuring MC-LAG or port channels on Dell PowerSwitches) remains out of scope and is the responsibility of the customer.

---

Requirements

Requirement 1: Active-Backup Bonding Provisioning (InfiniBand & Ethernet) [Req 1]

The system must allow administrators to define a logical bonded interface using active-backup mode (mode=1) across two or more physical ports (e.g., ib0, ib1 or eth1, eth2), regardless of whether those ports belong to a single double-port NIC or dual single-port NICs.

Relates to: Use Case 1

Scenario: Provisioning a Bond

• When the compute node is provisioned, the OS is configured such that bond0 is the active logical interface and physical link failure on the active slave results in immediate failover to the standby slave.

Requirement 2: Tagged VLAN Support on Bonds [Req 2]

The system must support the assignment of 802.1Q VLAN tags to a logical bonded interface.

Relates to: Use Case 2

Scenario: VLANs over Bond

• When the node is provisioned, the OS successfully creates the bond0.100 sub-interface, and traffic routed through it is correctly tagged and highly available.

---

Usability

Network configuration syntax in the Omnia inventory should be intuitive and align with standard Ansible/NetworkManager (or similar tool) data structures (e.g., specifying type: bond, mode: active-backup, and a list of slaves).

---

Reliability

Failure Mode	System Behavior
Misconfigured Slaves	If an administrator specifies a non-existent physical interface as a slave, the provisioning playbook must fail explicitly with a clear error message rather than silently leaving the node in a degraded state.

---

Telemetry

The network bonding configuration must expose metrics to track bond creation success rates, failover event frequency, and link status monitoring. Engineering should utilize Prometheus node_exporter's bonding collector or Telegraf's bond plugin to expose this data to the central observability stack.

---

Performance

Active-backup mode provides fault tolerance, not increased throughput. Throughput is limited to the bandwidth of a single active physical link.

---

Documentation

• Network Configuration Guide: Document the YAML syntax required to define a bonded interface in active-backup mode, including examples of applying VLANs on top of the bond. Emphasize that LACP (Mode 4) is not supported in this release.

---

Dependencies

• Physical Network: The physical Top-of-Rack switches must be configured to allow MAC address failover/flapping, as active-backup mode involves the standby NIC adopting the active NIC's MAC address upon failover. Port security features restricting MAC movement must be disabled on these ports.
• BR-54183 (Service Node Network Segmentation): Decoupled. Service nodes have shifted to a single-port shared LOM architecture (no bonding), making JA-8825 strictly applicable to the compute node fabric.
• Prerequisite: BR-36094 Req 4 (Admin Interface Boot Pinning). Bonded nodes expose multiple raw interfaces during PXE boot (before the OS configures the bond). Without deterministic boot interface selection, dracut may attempt DHCP on the standby port, causing boot failures.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Switch Port Security Blocking Failover	Operational	High	Medium	If switch port security prevents MAC flapping, the active-backup failover will be blocked by the switch, breaking network connectivity. Mitigate by explicitly documenting the required switch-side port configurations.

---

Acceptance Criteria

• AC-001: An administrator can define an active-backup bonded interface (mode=1) in the Omnia inventory for both InfiniBand (ibX) and Ethernet (ethX) interfaces. [Req 1]
• AC-002: Upon provisioning, the target node successfully configures NetworkManager (or a similar tool) to aggregate the specified physical ports into the bond, supporting both single double-port NICs and dual single-port NICs. [Req 1]
• AC-003: In a physical test, unplugging the active cable causes the standby cable to resume traffic automatically without dropping the storage mount. [Req 1]
• AC-004: Administrators can successfully provision tagged VLAN sub-interfaces (Ethernet) or PKeys (InfiniBand) on top of the bonded interface. [Req 2]

---

Scope & Sequencing

Can be implemented independently of other network provisioning features. Targeted for Omnia 2.0.

Sequencing Recommendation (vs. BR-54183):
(Note: These specs are fully decoupled. BR-54183 relies on single-port Shared LOM, while JA-8825 relies on bonding for the compute fabric.)

---

Assumptions / Open Questions

Assumptions

• A1: The customer networking team will properly configure switch ports to support active-backup MAC transitions.

Open Questions

• None.

---