Static IP for PowerVault Connectivity (Alternate Baremetal Slurm Controller)

Start with the storage_config.yml schema and the cloud-init template that runs nmcli on baremetal Slurm controllers; the issue names iscsi_nic_ip, iscsi_nic_netmask and iscsi_nic_gateway as new keys. Confirm the DHCP default still applies when those keys are absent (AC1.2, AC2.3). Done means the static settings apply and survive reboot (AC-002) and the acceptance criteria AC1.1 to AC2.2 pass. The body is cut off mid-sentence in its open-questions section, so check the full text before starting.

Static IP for PowerVault Connectivity (Alternate Baremetal Slurm Controller)

---

Value Proposition

Customer Value

Ensures continuous access to critical storage data with reliable PowerVault connectivity.

---

User Personas

• Infrastructure Administrator (Infra Admin): Configures and maintains the HPC cluster storage.

---

Use Cases

Use Case 1: Connect to PowerVault

A Infrastructure Administrator (Infra Admin) connects a baremetal Slurm controller to PowerVault using a static IP.

---

Scope Definition

Goals

• Enable static IP configuration for secondary NICs to connect to PowerVault.

Non-Goals

• Primary Kubernetes-based Slurm Controller connectivity (handled in BR-31052).

---

Requirements

Requirement 1: Static IP Configuration [Req 1]

The system must configure a static IP on the designated secondary NIC for PowerVault connectivity.

Scenario 1

• When the system boots with a valid static IP configuration, the secondary NIC uses the static IP.

Scenario 2

• When the system boots with an invalid static IP configuration, it logs an error and falls back to a safe state.

---

Security

N/A — This feature introduces no new security boundaries beyond standard Omnia RBAC/TLS.

---

Dependencies

• Secondary Network Interface Static IP Configuration (BR-82555)

---

Summary

The Infrastructure Administrator (Infra Admin) needs to configure stable, predictable network connectivity for iSCSI storage connections on baremetal Slurm controller nodes (which serves as the alternate/fallback configuration when a Kubernetes service cluster is not present). By adding post-boot static IP configuration for secondary network interfaces (iSCSI NIC), we eliminate DHCP-induced IP volatility that causes iSCSI connectivity loss, ensuring reliable PowerVault storage access with minimal engineering effort and backward compatibility.

---

Problem Statement

Currently, Omnia configures all network interfaces on baremetal Slurm controller nodes via DHCP only. While this works for primary management interfaces, iSCSI storage connectivity requires stable, predictable IP addresses. DHCP lease expiration or renewal can cause the iSCSI NIC to receive a new IP address, breaking active iSCSI sessions and causing storage connectivity loss. This forces administrators to manually reconfigure iSCSI connections or restart services, introducing operational risk and downtime. PowerVault storage systems require static IP configuration for reliable operation, making DHCP-only configuration unsuitable for production iSCSI deployments.

---

Telemetry

Monitoring Script (Sample)

# Check gateway reachability
if ping -c 1 -W 2 "$ISCSI_GATEWAY" >/dev/null 2>&1; then
  echo "iscsi_gateway_reachable 1" | nc -w 1 localhost 8125
else
  echo "iscsi_gateway_reachable 0" | nc -w 1 localhost 8125
fi

# Check target reachability
if ping -c 1 -W 2 "$ISCSI_TARGET" >/dev/null 2>&1; then
  echo "iscsi_target_reachable 1" | nc -w 1 localhost 8125
else
  echo "iscsi_target_reachable 0" | nc -w 1 localhost 8125
fi

# Check iSCSI session status
if iscsiadm -m session 2>/dev/null | grep -q "$ISCSI_TARGET"; then
  echo "iscsi_session_active 1" | nc -w 1 localhost 8125
else
  echo "iscsi_session_active 0" | nc -w 1 localhost 8125
fi

Phase 2 Alerting Rules (Future):

• Alert if iscsi_gateway_reachable = 0 for >5 minutes
• Alert if iscsi_session_active = 0 for >2 minutes
• Alert if iscsi_target_reachable = 0 for >5 minutes

---

Documentation

Example Configuration

The following values are illustrative and follow the PowerVault deployment-guide pattern. Production values are supplied by the deployment owner. Redundant deployments normally place storage paths on separate example subnets such as 10.10.10.0/24 and 10.10.20.0/24.

powervault_config:
 ip: 10.10.10.200
 iscsi_nic_ip: 10.10.10.20
 iscsi_nic_netmask: 255.255.255.0
 iscsi_nic_gateway: 10.10.10.1

Troubleshooting Steps

1. Verify iSCSI NIC IP address using ip addr show
2. Check gateway reachability using ping -c 1 -W 2 <gateway_ip>
3. Verify iSCSI target reachability using ping -c 1 -W 2 <target_ip>
4. Check iSCSI session status using iscsiadm -m session

---

Usability

Configuration Simplicity

Aspect	Requirement	Rationale
Configuration File	storage_config.yml	Simple, human-readable YAML format.
Parameter Names	iscsi_nic_ip, iscsi_nic_netmask, iscsi_nic_gateway	Clear, descriptive names for easy understanding.

Error Messaging

Aspect	Requirement	Rationale
Invalid IP Address	Error message indicates invalid IP address format.	Helps administrators quickly identify and fix configuration issues.
Unreachable Gateway	Error message indicates gateway is unreachable.	Enables administrators to diagnose and resolve connectivity issues.

---

Acceptance Criteria

• AC-001: Omnia user-data accurately processes static IP overrides [Req 1]
• AC-002: Baremetal node iSCSI NIC retains static IP after reboots [Req 1]
• AC-003: iSCSI initiator successfully discovers target on baremetal [Req 2]
• AC-004: iSCSI target multipath connections persist across node reboots [Req 2]

Requirement 1: Storage Configuration Schema Extension [Req 1]

AC ID	Description	Expected Result
AC1.1	storage_config.yml accepts new keys: iscsi_nic_ip, iscsi_nic_netmask, iscsi_nic_gateway	Schema validation passes when parameters are provided.
AC1.2	Schema validation passes when parameters are omitted (DHCP default behavior).	Schema validation passes when parameters are omitted.
AC1.3	Configuration is documented in Omnia documentation with examples for both DHCP and static IP modes.	Documentation includes examples for both DHCP and static IP modes.

Requirement 2: cloud-init Template Modification [Req 2]

AC ID	Description	Expected Result
AC2.1	cloud-init template detects presence of static IP parameters.	cloud-init template applies static IP configuration when parameters are present.
AC2.2	When parameters are present, nmcli commands set ipv4.method manual and configure IP, netmask, gateway, and DNS.	nmcli commands set ipv4.method manual and configure IP, netmask, gateway, and DNS.
AC2.3	When parameters are absent, nmcli defaults to ipv4.method auto (DHCP).	nmcli defaults to ipv4.method auto (DHCP) when parameters are absent.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Incorrect Static IP Configuration	Operational	iSCSI connectivity loss if IP is already in use or gateway is unreachable.	Medium	Validation script checks IP availability and gateway reachability before applying configuration. Documentation includes IP planning guidelines.
DHCP Server Conflict	Network	If DHCP server is still active, it may assign conflicting IP to another device.	Low	Documentation recommends disabling DHCP for iSCSI NIC or using DHCP reservation. Monitoring alerts on IP conflicts.
NetworkManager Upgrade Compatibility	Upstream Dependency	Future NetworkManager versions may change nmcli syntax or behavior.	Low	cloud-init template uses stable nmcli API; regression testing covers NetworkManager version compatibility.
iSCSI Session Persistence	Operational	If iSCSI NIC loses connectivity, active sessions may not auto-reconnect.	Low	Monitoring script detects disconnection; support runbook includes manual reconnection steps. Phase 2 will add auto-reconnect logic.
Configuration Drift	Operational	Manual nmcli commands post-deployment may override cloud-init configuration.	Medium	Documentation emphasizes idempotent configuration; support runbook includes steps to verify and restore configuration.
Multi-NIC Complexity	Design	Supporting multiple iSCSI NICs in future phases may introduce complexity.	Low	Phase 1 focuses on single NIC; Phase 2 will address multi-NIC MPIO with separate design.

---

Telemetry

Metric	Type	Collection	Target	Rationale
Static IP Adoption	Adoption	Count of clusters with static IP enabled	>50% of new deployments within 6 months	Measures feature adoption and customer demand.
iSCSI Connectivity Stability	Effectiveness	Count of iSCSI disconnections per cluster per month	<1 disconnection per month	Validates that static IP reduces iSCSI connectivity loss.
Configuration Errors	Quality	Count of invalid static IP configurations detected during deployment	<5% of deployments	Identifies usability issues with configuration parameters.
Support Tickets	Support Burden	Count of iSCSI connectivity-related support tickets	50% reduction within 6 months	Validates that feature reduces support burden.

---

APIs

N/A - This feature is configuration-file-based (storage_config.yml) and does not introduce new APIs. Configuration is applied via cloud-init template modifications and nmcli commands. No new REST APIs, gRPC services, or programmatic interfaces are introduced in Part 1.

---

Scope & Sequencing

Alternate Configuration: Baremetal Slurm Controller (This Spec)

• Static IP configuration for single iSCSI NIC
• DHCP remains default (opt-in static IP)
• Manual IP specification required
• RHEL 10.x support
• Custom monitoring script (Phase 1)
• Backward compatible

Primary Configuration: Kubernetes-Based Slurm Controller (BR-31052)

• Kubernetes-native networking approach
• Service VIP or DNS-based discovery
• Potential IPAM integration
• Multi-NIC MPIO support
• Enhanced monitoring and alerting
• Separate specification (BR-31052)

Non-Goals (Deferred to Phase 2 or Future)

• Multiple iSCSI NICs with MPIO
• Per-NIC DNS configuration
• Automated IP address allocation (IPAM)
• Kubernetes-based static IP
• Network interface bonding

---

Answers to Open Questions (Based on Industry Best Practices)

1. iSCSI NIC Identification

Answer: Use MAC address as primary identifier, with optional interface name as secondary fallback.

Rationale: MAC address is hardware-specific and persists across reboots and network reconfigurations. Interface names (eth0, eno1, etc.) can change based on kernel drivers and BIOS settings, making them unreliable for long-term identification.

Implementation: storage_config.yml includes iscsi_nic_mac_address as primary key; cloud-init template uses nmcli to identify NIC by MAC and apply configuration.

---

2. Static IP Persistence Mechanism

Answer: Use NetworkManager persistent configuration files (/etc/NetworkManager/system-connections/).

Rationale: NetworkManager is the standard network management service on RHEL 10.x. Persistent configuration files survive reboots, DHCP lease cycles, and network service restarts. This is the recommended approach per Red Hat documentation.

Implementation: cloud-init template uses nmcli commands to create persistent NetworkManager connections with static IP settings.

---

3. iSCSI Connectivity Validation

Answer: Validate gateway reachability and iSCSI target discoverability during cloud-init execution.

Rationale: Gateway reachability confirms network connectivity; iSCSI target discoverability confirms storage accessibility. Both must be validated to ensure the feature works end-to-end.

Implementation: cloud-init template includes validation steps using ping and iscsiadm commands; deployment fails with clear error messages if validation fails.

---

4. Backward Compatibility Strategy

Answer: Make static IP parameters optional in storage_config.yml; DHCP remains default.

Rationale: Existing clusters and configurations must not be affected. Static IP is an opt-in feature enabled only when parameters are explicitly provided.

Implementation: cloud-init template conditionally applies static IP configuration only when parameters are present; otherwise defaults to DHCP.

---

5. Multi-NIC Support in Phase 1

Answer: Single iSCSI NIC only in Phase 1; multi-NIC MPIO deferred to Phase 2.

Rationale: Single NIC keeps Phase 1 focused and achievable. Customer demand is for single NIC stability, not redundancy. Multi-NIC MPIO adds complexity and requires separate design.

Implementation: storage_config.yml supports single iscsi_nic_mac_address in Phase 1; Phase 2 will extend to multiple NICs.

---

6. DNS Configuration for iSCSI NIC

Answer: Optional DNS configuration via iscsi_nic_dns parameter; iSCSI uses IP addresses, not DNS names.

Rationale: iSCSI initiators connect to target IP addresses, not DNS names. DNS configuration is optional for administrative convenience (e.g., resolving PowerVault hostname to IP). Not required for iSCSI connectivity.

Implementation: cloud-init template conditionally applies DNS settings if provided; iSCSI connectivity does not depend on DNS.

---

7. Monitoring and Alerting Approach

Answer: Custom monitoring script in Phase 1 exposing metrics to StatsD; enhanced alerting in Phase 2.

Rationale: Phase 1 focuses on basic connectivity monitoring. Phase 2 will integrate with Omnia monitoring stack for advanced alerting and dashboards.

Implementation: Custom script runs every 5 minutes, exposes 4 metrics (IP address, gateway reachability, target reachability, session status) to StatsD on localhost:8125.

---

8. OS Compatibility

Answer: RHEL 10.x only in Phase 1; Ubuntu/Debian support is out of scope for the current Omnia release.

Rationale: Omnia 2.2 targets RHEL 10.0 and later. nmcli and NetworkManager are the supported configuration tools on RHEL 10.x.

Implementation: Phase 1 uses nmcli for RHEL 10.x; no Ubuntu/Debian configuration is included.

---

Deployment Runbook (Sample)

Step 1: Configure storage_config.yml

powervault_config:
 ip: 10.10.10.200
 iscsi_initiator: iqn.2025-01.com.example:scontrol
 volume_id: <example-storage-volume-id>
 iscsi_nic_mac_address: "<example-storage-nic-mac>"
 iscsi_nic_ip: 10.10.10.20
 iscsi_nic_netmask: 255.255.255.0
 iscsi_nic_gateway: 10.10.10.1

Step 2: Deploy Slurm Controller Node

ansible-playbook -i inventory.yml deploy-slurm-controller.yml

Step 3: Verify Static IP Configuration

# Check NetworkManager connection

nmcli con show | grep -A 5 "iscsi"

# Verify IP address

ip addr show | grep 10.10.10.20

# Check gateway reachability

ping -c 1 10.10.10.1

Step 4: Validate iSCSI Connectivity

# Discover iSCSI target

iscsiadm -m discovery -t sendtargets -p 10.10.10.200

# Login to iSCSI target

iscsiadm -m node -T iqn.2025-01.com.example:powervault -p 10.10.10.200 -l

# Verify session is active

iscsiadm -m session

---

Troubleshooting Guide (Sample)

Issue: iSCSI NIC Does Not Have Static IP

Symptoms: ip addr show does not show 10.10.10.20 on iSCSI NIC

Diagnosis:

nmcli con show <nic-name>
# Expected: ipv4.method: manual

# Expected: ipv4.addresses: 10.10.10.20/24

Resolution:

1. Verify storage_config.yml has correct static IP parameters
2. Re-run cloud-init: cloud-init clean --logs --seed && cloud-init init
3. Manually apply if needed: nmcli con modify <nic-name> ipv4.method manual ipv4.addresses 10.10.10.20/24 ipv4.gateway 10.10.10.1
4. Bring interface up: nmcli con up <nic-name>

Issue: Gateway Is Unreachable

Symptoms: cloud-init fails with "gateway unreachable" error

Diagnosis:

ping -c 1 10.10.10.1
# Expected: Responses from gateway

Resolution:

1. Verify network cable is connected to correct switch port
2. Verify gateway IP address in storage_config.yml is correct
3. Check switch VLAN configuration
4. Verify no firewall rules blocking ICMP
5. Check if gateway is on same subnet: ip route show

Issue: iSCSI Session Is Not Active

Symptoms: iscsiadm -m session shows no active sessions

Diagnosis:

iscsiadm -m discovery -t sendtargets -p 10.10.10.200
# Expected: Target discovery succeeds

Resolution:

1. Verify iSCSI NIC has static IP: ip addr show
2. Verify target IP is reachable: ping 10.10.10.200
3. Verify PowerVault is online and accessible
4. Check firewall rules allow port 3260: netstat -an | grep 3260
5. Verify iSCSI initiator name matches PowerVault ACL: cat /etc/iscsi/initiatorname.iscsi
6. Manually login: iscsiadm -m node -T iqn.2025-01.com.example:powervault -p 10.10.10.200 -l

---

Error Message Examples

Error: Gateway Unreachable

ERROR: iSCSI gateway 10.10.10.1 is not reachable from iSCSI NIC.
Verify network connectivity:
 1. Check network cable is connected
 2. Verify gateway IP address is correct in storage_config.yml
 3. Check switch configuration for VLAN/port settings
 4. Run: ping 10.10.10.1 from iSCSI NIC
 5. Check firewall rules: iptables -L -n | grep 10.10.10.1

Error: Static IP Already in Use

ERROR: Static IP 10.10.10.20 is already in use on the network.
Choose a different IP address:
 1. Scan network for available IPs: nmap -sn 10.10.10.0/24
 2. Update storage_config.yml with new IP
 3. Re-run deployment
 4. Verify no other nodes are using this IP: arp-scan 10.10.10.0/24

Error: iSCSI Initiator Cannot Discover Target

ERROR: iSCSI initiator cannot discover PowerVault target at 10.10.10.200.
Verify iSCSI configuration:
 1. Check target IP is correct: iscsiadm -m discovery -t sendtargets -p 10.10.10.200
 2. Verify PowerVault is online and accessible: ping 10.10.10.200
 3. Check firewall rules allow port 3260: netstat -an | grep 3260
 4. Verify iSCSI initiator name matches PowerVault ACL: cat /etc/iscsi/initiatorname.iscsi
 5. Check PowerVault ACL configuration for initiator IQN

Error: iSCSI Session Disconnection

ERROR: iSCSI session to 10.10.10.200 has disconnected.
Verify network stability:
 1. Check if gateway is still reachable: ping 10.10.10.1
 2. Check if target is still reachable: ping 10.10.10.200
 3. Verify static IP is still configured: ip addr show
 4. Check for network errors: dmesg | grep -i iscsi
 5. Check monitoring metrics: /usr/local/bin/iscsi-connectivity-monitor.sh
 6. Manually reconnect: iscsiadm -m node -T iqn.2025-01.com.example:powervault -p 10.10.10.200 -l

---

Assumptions / Open Questions

Assumptions

• A1: Required underlying hardware and OS capabilities are present.

Open Questions

• None.

---