Proxmox VE Infrastructure High Availability for Two OIM VMs

The payload names no files, tests or entry points in this repository; it is a specification for customer-managed Proxmox VE HA covering both OIM VMs. Start by answering the open questions it lists: supported Proxmox and pve-ha-manager configurations, the isolation-evidence format and freshness window, and the approved restart limit and interval. Done means those decisions are recorded and the acceptance criteria AC-001 to AC-008 can be checked against them.

Proxmox VE Infrastructure High Availability for Two OIM VMs

---

Summary

Infrastructure Administrators need the customer-managed Proxmox VE HA Manager (ha-manager / pve-ha-manager) to keep both BR-96665-delivered OIM VMs recoverable across VM and host failures. Proxmox Datacenter Manager (PDM) may provide centralized inventory, status, power operations, migration, and aggregated provider observability, but the Proxmox VE HA Manager is the automatic HA/failover engine. JA-14405 concurrently owns OIM service HA and service-level logs.

---

Problem Statement

Two-VM service HA cannot recover a VM or failed hypervisor host, while provider-level VM restart cannot determine whether OIM services, VIP, quorum, or shared state are ready. Treating either layer as the sole HA controller leaves a recovery gap.

The infrastructure layer must protect both OIM VMs, publish trustworthy isolation and recovery evidence, avoid reacting to service-only failures, and return a recovered VM for JA-14405 validation and passive rejoin.

---

Value Proposition

Customer Value

Customers retain two-VM OIM service continuity while customer-managed Proxmox VE HA Manager restores failed VM infrastructure. Operators can distinguish VM recovery from OIM service readiness and correlate both layers without Omnia controlling the virtualization provider.

---

User Personas

• Infrastructure Administrator: Owns Proxmox VE HA Manager VM liveness, placement, restart, isolation, and provider health.
• OIM Administrator: Owns OIM service HA through JA-14405 and needs recovered VM handoff.
• Site Reliability Engineer: Correlates VM-level and service-level observability across recovery events.

---

Use Cases

Use Case 1: Recover a Failed OIM VM

Actor: Infrastructure Administrator
Context: One OIM VM or its host becomes unavailable.
Action: Proxmox VE HA Manager isolates the failed instance, prevents duplicate execution, and restores the VM on eligible infrastructure.
Result: JA-14405 keeps services on the surviving VM and later evaluates the recovered VM for passive rejoin.

Use Case 2: Preserve Service-Level Ownership

Actor: OIM Administrator
Context: An OIM service fails while its VM remains healthy.
Action: JA-14405 handles service restart or failover and Proxmox VE HA Manager does not restart the VM.
Result: Service recovery and VM recovery remain independent and do not form a restart loop.

Use Case 3: Correlate VM and Service Recovery

Actor: Site Reliability Engineer
Context: A host failure triggers provider and service events.
Action: The engineer correlates VM identity, host placement, power state, isolation evidence, restart outcome, OIM role, VIP, quorum, and service readiness.
Result: VM running and OIM service ready remain separate observable states.

---

Scope Definition

Goals

• Protect both OIM VMs delivered through BR-96665.
• Monitor VM power/liveness, host placement, provider health, and restart state.
• Prevent duplicate VM execution and publish VM-specific isolation evidence.
• Restore a failed VM on eligible infrastructure and hand it back to JA-14405.
• Bound provider restart behavior and prevent cross-layer restart loops.
• Expose VM-level observability correlated with JA-14405 service events.

Non-Goals

• Monitoring, restarting, failing over, or declaring readiness of OIM services.
• Managing the OIM VIP, quorum, active/passive role, state consistency, or passive rejoin.
• Replacing JA-14405 service HA or JA-3733 upgrades and rollback.
• Omnia control of Proxmox VE HA Manager APIs or provider lifecycle operations.
• Protecting only one production OIM VM or replacing the two-VM topology.

---

Requirements

Requirement 1: Two-VM Infrastructure Protection [Req 1]

Customer-managed Proxmox VE HA Manager must independently monitor and protect both OIM VMs across eligible VM and host failures.

Scenario: Both VMs healthy

• VM identity, power state, host placement, and provider health are available for both VMs.

Scenario: One VM fails

• The failed VM is isolated and restored without restarting the surviving healthy VM.

Requirement 2: Duplicate Prevention and Isolation Evidence [Req 2]

The infrastructure layer must prevent duplicate execution and publish authenticated, fresh, VM-specific evidence that a failed VM cannot remain active.

Scenario: Isolation confirmed

• JA-14405 can consume the evidence before service promotion.

Scenario: Isolation uncertain

• Provider recovery and service promotion do not claim success and the missing evidence is reported.

Requirement 3: Full-VM Recovery and Handoff [Req 3]

The Proxmox VE HA Manager must restore a failed VM with its required virtual hardware, persistent VM storage, and network access and report it as VM-running separately from OIM readiness. Planned live migration may retain memory only while the source host remains operational; unplanned host loss cannot recover memory state and requires fencing plus VM restart from persistent storage on eligible infrastructure.

Scenario: VM restored

• The recovered VM is handed to JA-14405 for state, cluster, and service validation before passive rejoin.

Scenario: VM cannot recover

• Provider failure is reported while JA-14405 continues service on the surviving VM when safe.

Requirement 4: Restart-Loop Prevention [Req 4]

Proxmox VE HA Manager must respond to VM or host failure and must not restart a healthy VM solely because an OIM service failed.

Scenario: Service-only failure

• JA-14405 handles the failure and no provider VM restart occurs.

Scenario: VM or host failure

• Proxmox VE HA Manager handles VM recovery and JA-14405 does not request full VM restart.

Requirement 5: Bounded Provider Recovery [Req 5]

Provider restart attempts, intervals, terminal failure, and escalation must follow an approved bounded policy.

Scenario: Restart succeeds

• Restart count and outcome are recorded and the VM proceeds to JA-14405 validation.

Scenario: Restart limit reached

• Automated retry stops, escalation occurs, and the surviving service state is not disturbed.

Requirement 6: VM-Level Observability [Req 6]

The infrastructure layer must expose VM power/liveness, host placement, provider health, isolation state, restart attempts, restart outcome, recovery duration, and VM identity/timestamps for both OIM VMs.

Scenario: Healthy infrastructure

• Operators can identify both VM states, placements, and provider readiness.

Scenario: Cross-layer incident

• VM events correlate with JA-14405 service-level events without equating VM running with OIM ready.

---

Reliability

Failure Mode	System Behavior
OIM VM failure	Failed VM is isolated and restored without disturbing the surviving VM.
Planned host maintenance	Live migration may preserve VM memory while the source host remains operational.
Unplanned hypervisor-host failure	Memory state is lost; the HA Manager fences the failed host and restarts the VM from persistent storage on eligible capacity.
Duplicate execution risk	Recovery blocks until provider isolation is established.
Restart limit reached	Automated retry stops and operator escalation begins.
VM restored but OIM unhealthy	Infrastructure reports VM running while JA-14405 reports service not ready.

Recovery: VM-level completion means the guest is running with required virtual resources and reachability; OIM readiness remains a JA-14405 outcome.

---

Security

• Proxmox VE HA Manager ownership and credentials remain customer-managed.
• Isolation evidence is authenticated, fresh, and VM-specific.
• Provider logs and observability do not expose secrets.
• Omnia requires no provider lifecycle credentials under this specification.

---

Telemetry

BR-96666 uses provider-native Proxmox VE HA Manager status and logs for VM power, liveness, placement, fencing/isolation, restart attempts, outcomes, and recovery duration for both OIM VMs; PDM may aggregate and display that provider observability. JA-14405 separately sends service-level rsyslog events through Vector to VictoriaLogs, and VM running remains distinct from OIM service ready.

---

Documentation

• Proxmox VE HA Manager prerequisites and two-VM placement guidance.
• Isolation-evidence contract and cross-layer identity/timestamp model.
• Runbooks distinguishing planned live migration from unplanned fencing and persistent-storage restart.
• Provider-native HA status/log fields and optional PDM aggregation guidance.
• Observability guide distinguishing VM running from OIM service ready.
• Customer virtualization testbed construction and provider configuration remain customer-owned, not Omnia documentation.

---

Dependencies

• Two compatible OIM guests delivered through BR-96665.
• JA-14405 concurrent service HA and service-level observability.
• Customer-managed Proxmox VE cluster and HA Manager (pve-ha-manager, CRM/LRM, quorum, watchdog/locking/fencing, persistent storage, networking, and eligible placement); optional PDM provides centralized management and observability but is not the HA engine.
• JA-3733 for OIM upgrades and rollback.
• Authenticated cross-layer isolation and correlation evidence.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Provider and Pacemaker both restart	Operational	High	Medium	Enforce failure ownership and restart-loop prevention.
Isolation evidence is stale	Technical	High	Medium	Require authenticated freshness and VM identity.
VM running is reported as OIM ready	Operational	High	Medium	Keep provider and service readiness states separate.
Both VMs share one failure domain	Technical	High	Low	Validate placement before enabling recovery.
Restart storm affects healthy VM	Operational	High	Low	Use bounded per-VM restart policy and leave surviving VM untouched.

---

Acceptance Criteria

• AC-001: Proxmox VE HA Manager reports identity, power state, placement, and health for both OIM VMs. [Req 1] [Req 6]
• AC-002: Failure of one VM isolates and restores only that VM while the survivor remains available to JA-14405. [Req 1] [Req 3]
• AC-003: Fresh VM-specific isolation evidence is available before JA-14405 service promotion. [Req 2]
• AC-004: Missing isolation evidence blocks successful recovery claims. [Req 2]
• AC-005: A restored VM is reported as running and is handed to JA-14405 before passive rejoin. [Req 3]
• AC-006: Service-only failure does not cause provider VM restart, and VM failure does not cause JA-14405 to request restart. [Req 4]
• AC-007: Restart attempts stop and escalate at the approved bound without disturbing the surviving VM. [Req 5]
• AC-008: VM-level observability correlates with JA-14405 events while preserving separate VM-running and service-ready states. [Req 6]

---

Scope & Sequencing

BR-96666 is separately backlog-sequenced outside the initial Omnia 2.3 JA-14405 service-monitoring delivery without naming a speculative release. BR-96665 delivers two OIM guests, JA-14405 provides concurrent service HA, the customer-managed Proxmox VE HA Manager provides VM recovery, and JA-3733 provides upgrades and rollback.

---

Assumptions / Open Questions

Assumptions

• Both OIM VMs match the BR-96665 profile and occupy separate eligible failure domains.
• Proxmox VE HA Manager can publish trustworthy VM-specific isolation and recovery evidence.
• Provider recovery ignores service-only failures.

Open Questions

• Which Proxmox VE, pve-ha-manager, optional PDM, quorum, watchdog/fencing, placement, and persistent-storage configurations are supported?
• How is isolation evidence published and how fresh must it be?
• What restart attempt limit and interval are approved?
• What VM detection, isolation, and restart targets are approved?

---