Rolling Upgrades and Rollbacks for OIM and Compute Scalable Units

The issue names no files, tests or entry points, so start by finding where Omnia Ansible resolves compute SU membership from topology data and how the OIM HA roles are represented. Done would be a gated, resumable rollout that stops on membership or validation failure. This is design work across OIM HA and SU sequencing, so agree the scope with maintainers before writing code.

Rolling Upgrades and Rollbacks for OIM and Compute Scalable Units

---

Summary

HPC Infra Admins need a standardized way to upgrade and roll back Omnia without updating the entire cluster at once. The capability updates and validates the passive OIM before a controlled role transition, then updates and validates the former active OIM so both instances converge on the target version. Omnia Ansible automation also upgrades or rolls back customer-defined compute Scalable Units (SUs) sequentially, keeping unaffected SUs and their workloads available.

---

Problem Statement

Omnia does not currently provide one cohesive upgrade workflow across its management and compute layers. OIM upgrades need a validated passive-first sequence through both-instance convergence, while compute updates need workload-aware gates and rollback at customer-defined SU boundaries. Upgrade sequencing must also respect flexible functional groups and federated OpenCHAMI worker domains without assuming those boundaries are identical to an SU.

---

Value Proposition

Customer Value

Customers can update Omnia while work continues outside the OIM instance or SU being serviced. Administrators receive clear validation gates, progress visibility, and rollback outcomes while the rollout remains bounded to one management instance or compute SU at a time.

---

User Personas

• Infra Admin: Owns the OIM and infrastructure lifecycle. Needs controlled upgrades and rollbacks that preserve management availability.
• HPC Systems Engineer: Owns compute topology and workload operations. Needs SU-by-SU updates that respect functional-group and provisioning boundaries.

---

Use Cases

Use Case 1: Passive-First OIM Rolling Upgrade

Actor: Infra Admin
Context: A supported Omnia update is available for an HA OIM deployment.
Action: The administrator updates and validates the passive OIM, performs a controlled role transition, and then updates and validates the former active OIM in its passive role.
Result: Both OIM instances converge on the target version while one stable OIM service remains available.

Use Case 2: Compute SU Rolling Upgrade

Actor: HPC Systems Engineer
Context: A supported Omnia-managed update must be applied across multiple compute SUs.
Action: Omnia Ansible automation prepares, updates, validates, and returns one SU to service before proceeding to the next.
Result: Unaffected SUs remain available, and functional-group and federated-provisioning boundaries remain intact.

Use Case 3: Compute SU Rollback

Actor: HPC Systems Engineer
Context: An update or post-update validation fails within the current SU.
Action: The administrator invokes the supported rollback for that SU while later SUs remain unchanged.
Result: The affected SU returns to its prior supported state, and the rollout remains paused until the failure is resolved.

---

Scope Definition

Goals (Execution Outcomes)

• Complete a passive-first OIM upgrade through validation, controlled role transition, and target-version convergence on both OIM instances.
• Provide controlled OIM rollback without permitting two active instances or incompatible shared state.
• Use Omnia Ansible automation for graceful rolling upgrades and rollbacks across customer-defined compute SUs.
• Keep unaffected SUs available and preserve each flexible functional group's software, image, and boot assignments.
• Respect federated OpenCHAMI worker boundaries and stop progression when topology, readiness, update, or validation checks fail.

Non-Goals (Scope Boundaries)

• Defining the OIM HA mechanism; supported HA profiles provide role identity, controlled transition, state availability, and recovery behavior.
• Creating software packages, operating-system images, or firmware payloads.
• Updating hardware firmware.
• Redefining flexible functional groups, federated OpenCHAMI topology, or workload scheduling policy.
• Updating all compute SUs concurrently or requiring destructive cluster-wide reprovisioning.

---

Requirements

Requirement 1: Passive-First OIM Rolling Upgrade [Req 1]

The system must update and validate the passive OIM before a controlled role transition, then update and validate the former active OIM in its passive role.

Scenario: OIM Upgrade Convergence

• Before changing either instance, the system confirms active and passive roles and validates HA and shared-state readiness.
• The passive OIM passes required service, state, security, and connectivity checks before it can receive the active role.
• After the role transition, the former active OIM is updated and tested as passive.
• The upgrade completes only when both instances run the target version and the stable service endpoint is healthy.

Scenario: Passive OIM Validation Failure

• When the updated passive OIM fails a required check, the current active OIM remains in service, role transition is blocked, and the upgrade does not advance.

Requirement 2: OIM Rolling Rollback [Req 2]

The system must provide a supported rollback path for failures before role transition, after role transition, and after both OIM instances converge.

Scenario: Rollback

• The system restores service through the last validated compatible OIM state and verifies the stable endpoint before rollback completes.
• If both instances reached the target version, both return to a supported prior state through a controlled sequence with state compatibility validation.

Requirement 3: Compute SU Rolling Upgrade [Req 3]

The system must use Omnia Ansible automation to upgrade one customer-defined compute SU at a time while maintaining service for unaffected SUs. This capability owns deterministic rollout membership: every targeted node maps to exactly one SU. Membership is resolved from authoritative Omnia topology. An SU may contain multiple flexible functional groups and may align with a federated OpenCHAMI worker domain, but those boundaries are not assumed to be identical. Per-group and per-node approved software, image, and boot assignments remain authoritative within the selected SU.

Scenario: Successful SU Progression

• The system resolves SU membership, allows active work to complete or move, prevents new work from entering the SU, and consumes the final readiness result before making changes.
• It applies each node's approved update according to its functional-group and topology assignments, validates nodes and required services, and returns the SU to service before proceeding.
• Nodes and SUs outside the current upgrade boundary remain available.

Scenario: Topology or Readiness Failure

• When membership is missing, ambiguous, or inconsistent with functional-group or federated-provisioning boundaries, the upgrade stops before changing nodes and identifies the affected topology data.

Requirement 4: Compute SU Rollback and Resume [Req 4]

The system must roll back the affected SU when an update or required validation fails and preserve enough progress information to resume safely.

Scenario: SU Update Failure or Interrupted Rollout

• Later SUs remain unchanged, the rollout pauses, and the affected SU returns to its prior supported state before regaining workload eligibility.
• On resume, the system identifies completed, rolled-back, and pending SUs without repeating a completed SU unless explicitly selected.

---

Reliability

Failure Mode	System Behavior
Passive OIM update or validation fails	The active OIM remains in service and role transition is blocked.
Planned OIM role transition fails	The last validated compatible OIM retains or regains the service role before the upgrade continues.
Former active OIM fails to converge	The newly active OIM remains in service while the passive instance is recovered or rolled back.
Current SU fails upgrade or validation	Later SUs remain unchanged, the rollout pauses, and the current SU follows its rollback path.
Upgrade topology changes during execution	The rollout pauses until authoritative SU membership is refreshed.

Availability: The stable OIM service remains available through the planned role transition, and compute capacity outside the current SU remains available.
Recovery: Rollback restores a validated supported state at the OIM or current-SU boundary before progression resumes.

---

Telemetry

The system reports OIM upgrade phase, passive validation, role transition, version convergence, current SU, completed and pending SUs, readiness, failure point, and rollback status to the standard observability stack.

---

Security

• Upgrade and rollback artifacts must come from approved, authenticated sources and remain verifiable throughout the supported rollback window.
• Existing access controls, secrets handling, and certificate trust must remain valid before an OIM instance or SU returns to service.
• Upgrade records must not expose credentials or secret material.

---

Documentation

• Admin Guide: Document supported upgrade paths, prerequisites, SU selection, service continuity, and completion evidence.
• Runbooks: Document passive-first OIM upgrade and rollback, planned role transition, SU-by-SU progression, failed-gate recovery, and safe resume.
• Architecture Docs: Describe how the workflow consumes OIM HA roles, flexible functional groups, and federated OpenCHAMI topology.

---

Dependencies

• JA-14405: Provides the initial active/passive OIM roles, stable service endpoint, state availability, and controlled role transition.
• JA-14425: Provides operation-readiness checks for OIM health, provisioning, storage, and workloads.
• BR-21709: Provides independent flexible-functional-group software and image assignments.
• BR-80708: When deployed, provides federated OpenCHAMI worker boundaries that the SU sequence must respect.
• BR-96665 and BR-96666: The current one-production-VM profile does not satisfy the OIM rolling-upgrade requirements. The initial supported OIM path uses JA-14405 unless a separately approved Proxmox upgrade profile supplies two compatible upgrade environments and equivalent outcomes.
• Supported Update Artifacts: Approved source, target, and rollback artifacts remain available until convergence is accepted.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
OIM shared-state incompatibility	Technical	High	Medium	Block role transition or rollback when the selected state cannot be consumed safely.
Passive OIM validation is incomplete	Operational	High	Medium	Prevent transition until every mandatory validation outcome passes.
SU membership is ambiguous	Operational	High	Medium	Resolve membership from authoritative topology and stop on missing or conflicting mappings.
Active workloads are disrupted	Operational	High	Medium	Require workload-readiness gates and keep unaffected SUs available.
Partial rollout leaves mixed versions	Technical	Medium	Medium	Persist progress by OIM and SU boundary and provide rollback or safe resume.

---

Acceptance Criteria

• AC-001: The passive OIM reaches the target version and passes all required checks before a planned role transition. [Req 1]
• AC-002: The planned transition preserves exactly one reachable active OIM service identity, and the former active OIM is upgraded and validated as passive so both instances converge. [Req 1]
• AC-003: OIM rollback succeeds before and after convergence, restoring a supported compatible state and validating the stable service endpoint. [Req 2]
• AC-004: Omnia Ansible automation upgrades one authoritative SU at a time and starts the next only after the current SU passes all gates. [Req 3]
• AC-005: Workloads and compute capacity outside the current SU remain available. [Req 3]
• AC-006: A failed SU stops progression, leaves later SUs unchanged, and returns to its prior supported state before workload eligibility is restored. [Req 4]
• AC-007: An interrupted compute rollout resumes from recorded SU progress without losing functional-group or federated-topology boundaries. [Req 3, Req 4]

---

Scope & Sequencing

The initial OIM rolling-upgrade delivery follows JA-14405 because it requires two compatible OIM environments and a supported active/passive role contract. JA-14425 and BR-21709 provide reusable readiness and group scope before compute SU rollout. BR-80708 is conditional when federated OpenCHAMI is deployed. The current BR-96665 and BR-96666 one-production-VM profile does not satisfy the OIM rolling-upgrade requirements.

---

Assumptions / Open Questions

Assumptions

• A1: Each node maps to one authoritative SU for a rollout, even when flexible functional groups or federated OpenCHAMI boundaries overlap that SU.
• A2: The selected OIM HA profile can identify active and passive instances, validate the passive instance, perform a planned role transition, and preserve one stable service identity.
• A3: Supported releases define a compatibility and rollback window long enough to complete OIM and SU convergence.

Open Questions

• Q1: Which topology source is authoritative for SU membership when an SU does not align exactly with a functional group or federated OpenCHAMI worker domain?
• Q2: Which approved update artifact types are included in the first compute SU rollout profile?

---