Secrets Manager Integration for Host and iDRAC Credentials

This is a design spec and names no existing source files, tests or entry points, so start by asking the maintainers which module should own the secrets-manager client and how the bootstrap and operational iDRAC credentials are split. Assumptions A3 and A4 and the BR-31940 dependency are still unresolved. Done means AC-001 to AC-004 pass, and that work needs maintainer agreement on the design first.

Secrets Manager Integration for Host and iDRAC Credentials

Summary

Scaling hardware management to thousands of compute nodes requires non-default iDRAC (BMC) passwords and other lifecycle secrets. Storing highly privileged credentials within the management stack creates a broad security exposure if the management node is compromised. This capability integrates Omnia with an external secrets manager to retrieve required provisioning secrets during bare-metal lifecycle operations and then use low-privilege operational credentials for day-to-day access where needed.

---

Problem Statement

Deploying a massive HPC system (1,000+ servers) requires automated mechanisms to manage out-of-band hardware and provisioning secrets. Previously, Omnia managed a single uniform iDRAC password for the entire cluster stored locally, which provides the bare minimum of security.

Deployments may require group-level, non-default credentials for hardware and provisioning workflows. Privileged hardware passwords are provisioning secrets with a broad impact if exposed, and storing them inside the management stack violates the Principle of Least Privilege. Administrators require an external secrets manager while still allowing Omnia to automate initial node provisioning and satisfy stringent secrets-management and audit requirements.

---

Value Proposition

Customer Value

Sites can use an existing secrets manager to centrally manage and rotate administrative hardware credentials and related provisioning secrets while preserving their security boundaries during automated cluster provisioning.

---

User Personas

• Enterprise Security Administrator: Manages the external secrets manager, defines access policies, and grants short-lived read access to the Omnia cluster.
• HPC Systems Engineer: Uses Omnia to deploy the cluster, relying on Omnia's ability to act as a seamless broker between the external secrets manager and the target hardware.

---

Use Cases

Use Case 1: Provisioning Secret Retrieval (Bare-Metal Provisioning)

Actor: Omnia Provisioning Engine (Automated)
Context: Omnia is executing a PXE boot or Redfish configuration on a group of new PowerEdge servers.
Action: Using an API token, Omnia authenticates to the external secrets manager. It fetches the required provisioning secret for the target server group and uses it in memory to execute the configuration commands.
Result: The servers are successfully configured, and the provisioning secret is excluded from persistent storage and logs.

Use Case 2: Low-Privilege Fallback (Day-2 Telemetry)

Actor: Omnia Telemetry Services (vmagent / Exporters)
Context: Omnia needs to continuously poll the iDRACs for power and thermal metrics.
Action: During initial provisioning, Omnia uses an authorized bootstrap credential to establish a least-privilege management identity on the iDRAC. The bootstrap credential is not persisted after use, and the operational credential is stored only in an approved encrypted internal secrets manager with access control, rotation, audit, and retention policies.
Result: The telemetry services use the localized, low-privilege operational credential continuously. If compromised, the attacker can only read metrics, not wipe or power off the server.

---

Scope Definition

Goals (Execution Outcomes)

• Implement a secrets-manager client integration within Omnia to communicate with an external secrets manager.
• Allow Omnia to authenticate to the external secrets manager using a supported short-lived authentication mechanism.
• Fetch required provisioning secrets dynamically into Ansible execution memory during lifecycle tasks.
• Create and deploy low-privilege operational credentials to target hardware when continuous access is needed.

Non-Goals (Scope Boundaries)

• Omnia will NOT persistently cache, sync, or store privileged secrets on the local filesystem or inside its internal database.
• Omnia will NOT deploy, operate, or manage the external secrets manager. The deployment owner is responsible for operating that service.

---

Requirements

Requirement 1: External Secrets-Manager Authentication [Req 1]

The system must support authentication to an external secrets manager.

Relates to: Use Case 1

Scenario: External Secrets-Manager Authentication

• When the bare-metal provisioning workflow begins, the Omnia Infrastructure Manager establishes a secure connection to the secrets manager API and verifies its authorized read access for the required provisioning secrets.

Requirement 2: In-Memory Secret Retrieval [Req 2]

The system must ensure retrieved secrets are used strictly in memory and are never logged or stored.

Relates to: Use Case 1

Scenario: Ansible Execution Isolation

• When Ansible executes lifecycle modules, secrets are treated as sensitive variables (no_log: true), preventing them from appearing in stdout, syslog, or Omnia's internal state files.

Requirement 3: Operational Credential Tiering [Req 3]

The system must transition to localized, low-privilege credentials for continuous operations.

Relates to: Use Case 2

Scenario: Day-2 Role Creation

• When configuring the BMC, Omnia creates an explicitly scoped, low-privilege operational user on the iDRAC and stores its credential in the approved encrypted internal secrets manager with the required access, rotation, audit, and retention controls.

---

Usability

The configuration must identify the external secrets-manager endpoint, protected authentication reference, and authorized provisioning-secret reference without embedding credentials.

---

Reliability

Failure Mode	System Behavior
Secrets Manager Unreachable	If Omnia cannot reach the external secrets manager during provisioning, the process must immediately fail with an explicit error detailing the connection failure, preventing partially-configured nodes.

---

Telemetry

The secrets manager integration must expose metrics through a supported metrics endpoint to track authentication success rates, secret retrieval patterns, and credential tiering effectiveness.

---

Security

• All communication between Omnia and the external secrets manager must use TLS.
• The principle of credential tiering must be preserved: privileged/root credentials stay external, limited-scope operational credentials stay internal.

---

APIs

API	Type	Purpose	Consumers
Secrets Manager API	External Dependency	Provides the endpoints to fetch required secrets.	Omnia Provisioning Engine

---

Documentation

• Security Architecture Guide: Document the external secrets-manager integration boundary, client behavior, in-memory secret handling, access-policy expectations, and exclusion of secrets from persistent storage and logs.

---

Dependencies

• External Secrets Manager: The deployment owner must operate the external secrets manager; Omnia is a client of that service.
• BR-31940 (Secure CA Certificate Distribution): If the external secrets manager uses an internal Enterprise CA, nodes must have the CA trust chain in place before Omnia can establish a TLS connection to the secrets manager endpoint.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Token Expiration	Operational	High	Medium	If the Omnia API token expires during a massive 1,000+ node provisioning run, the deployment will fail halfway. Mitigate by implementing automated token renewal mechanisms or clearly documenting the required token TTL for scale-out runs.

---

Acceptance Criteria

• AC-001: Omnia successfully connects to an external secrets manager using an API token and retrieves a stored provisioning secret. [Req 1]
• AC-002: Bare-metal provisioning tasks succeed using the dynamically fetched secret, and verification confirms the secret is NOT written to any local Omnia file, database, or log. [Req 2]
• AC-003: A least-privilege operational identity is created on the target iDRAC during provisioning, its credential is stored in the approved encrypted internal secrets manager, and the bootstrap credential is not retained. [Req 3]
• AC-004: Day-2 telemetry services successfully authenticate to the target iDRAC using the internal low-privilege credential rather than the root credential. [Req 3]

---

Scope & Sequencing

Secrets-manager authentication must be integrated at the very beginning of the bare-metal/OIM initialization sequence.

---

Assumptions / Open Questions

Assumptions

• A1: The deployment owner operates the external secrets manager and defines the required access policies.
• A2: The architecture assumes the OIM communicates directly with the external secrets-manager API. If high-availability caching is required, the deployment owner configures an approved local proxy and provides its endpoint to Omnia.
• A3 (HA Interaction): If the OIM is deployed in a High-Availability cluster (JA-14405), the secrets-manager client must be capable of re-authenticating or securely transferring the token state during an active/passive failover event to ensure interrupted playbooks can resume.
• A4 (Virtualization Interaction): If the OIM is deployed as a virtual machine, the supported secrets-manager authentication mechanism must establish identity for the virtualized OIM appliance rather than relying only on bare-metal MAC or IP identity.

Open Questions

• None.

---