Secure CA Certificate Distribution Mechanism for Node Provisioning

This is a design-level feature spanning the OIM control plane, OpenCHAMI cloud-init provisioning and the RHEL trust store. The issue names omnia_config.yml for a proposed custom_ca_certs key and the trust path /etc/pki/ca-trust/source/anchors/, but no existing file. Start by finding where omnia_config.yml is parsed and where node cloud-init is generated. Done means a PEM CA supplied on the OIM is present and trusted on a freshly provisioned node, with an invalid PEM failing pre-flight.

Secure CA Certificate Distribution Mechanism for Node Provisioning

Summary

In strict Zero-Trust enterprise environments, cluster nodes must establish secure TLS connections to external resources (secrets managers, telemetry services, and container registries) immediately upon booting. Currently, nodes deployed by OpenCHAMI boot with only standard public OS trust stores. This capability introduces a secure, automated mechanism to inject site-provided Enterprise Certificate Authority (CA) root and intermediate certificates into the bare-metal compute nodes during the Day 1 provisioning phase via cloud-init or Golden Image configurations. This ensures that the node implicitly trusts internal enterprise infrastructure without relying on insecure workarounds like --insecure-skip-tls-verify.

---

Problem Statement

Modern HPC and AI clusters rely on a web of external dependencies. During the provisioning sequence, a node might need to authenticate to an external secrets manager, pull a container image from an internal private registry, or register itself with an internal mTLS telemetry service. Because the freshly imaged bare-metal node lacks the enterprise's custom PKI/CA certificates, these TLS handshakes fail.

Administrators currently must either disable TLS verification entirely (a massive security violation leading to failed compliance audits) or manually run post-deployment Ansible scripts to inject the CAs (a race condition, as Day 1 services like kubelet or vmagent might fail to start before the CA is injected).

---

Value Proposition

Customer Value

• Zero-Trust Automation: Customers achieve fully automated, unattended bare-metal provisioning that adheres to their internal Enterprise PKI policies from the very first boot sector.
• Reliability: Eliminates race conditions and deployment failures caused by untrusted certificate errors during automated Day-1 service startup.

---

User Personas

• Enterprise Security Administrator: Provides the Enterprise Root/Intermediate CA certificates (usually PEM encoded) that need to be trusted by the cluster.
• Omnia Infrastructure Administrator (Infra Admin): Supplies the CA files in the Omnia control plane prior to running the deployment, ensuring all downstream OpenCHAMI-provisioned nodes inject them properly.

---

Use Cases

Use Case 1: Early-Boot Trust for External Secrets Management

Context: A freshly imaged bare-metal node needs to retrieve an authorized provisioning value from an approved external secrets service.
Action: During OpenCHAMI provisioning, cloud-init injects the enterprise CA into /etc/pki/ca-trust/source/anchors/ and updates the system store.
Result: When the secrets-service client starts moments later, the TLS handshake to secrets.example.com succeeds securely.

Use Case 2: Trusting Private Container Registries

Context: A Kubernetes node needs to pull an NVIDIA NGC AI container from an internal caching registry (e.g., Harbor or Artifactory).
Action: The node boots with the custom CA already trusted via the Golden Image or early-boot provisioning pipelines.
Result: Container runtime (containerd/crio) successfully verifies the registry's TLS certificate and pulls the image, avoiding "x509: certificate signed by unknown authority" errors.

---

Scope Definition

Goals (Execution Outcomes)

• Provide a standardized configuration block in omnia_config.yml (e.g., custom_ca_certs: [path/to/certs]) for admins to supply internal CA certificates.
• Automate the injection of these certificates into the OS trust store (e.g., via update-ca-trust on RHEL/Rocky) during the standard node boot sequence.
• Ensure the CA distribution happens before critical cluster services (secrets-manager clients, Kubelet, and telemetry agents) attempt external connections.

Non-Goals (Scope Boundaries)

• Omnia will NOT act as a Certificate Authority (CA) itself to issue these root certificates.
• Omnia will NOT handle automated rotation or revocation of the Root CA certificates; this is purely for static distribution of the public trust anchors.

---

Requirements

Requirement 1: User-Provided CA Integration [Req 1]

Omnia must support ingesting one or more user-provided PEM-encoded CA certificates from the OIM (Control Plane) and passing them into the OpenCHAMI provisioning pipeline.

Requirement 2: Early Boot Injection (cloud-init) [Req 2]

The system must utilize cloud-init (write_files or ca-certs modules).

Requirement 3: OS Compatibility [Req 3]

The injection mechanism must correctly map to the RHEL 10.x trust store path (e.g., /etc/pki/ca-trust/source/anchors/). Ubuntu/Debian support is out of scope.

---

Reliability

Failure Mode	System Behavior
Invalid Certificate Format	Omnia pre-flight validation must fail early if the provided CA file is not a valid PEM-encoded certificate, preventing nodes from deploying with broken trust stores.

---

Security

• This mechanism only distributes public CA certificates. At no point should private keys associated with the CA be distributed or requested.

---

Dependencies

Cross-Cutting Platform Dependencies

• IPv6 Compatibility: CA certificate injection via cloud-init must work over IPv6 provisioning paths (BR-18602). Certificate distribution endpoints must be reachable via IPv6.
• OIM HA Awareness: The CA root key material resides on the OIM. In an HA configuration (JA-14405), the CA state must replicate to the passive node to survive failover.
• Downstream dependents: This is a critical foundational dependency for: federated cert chain (BR-80708 Req 6), mTLS telemetry (JA-17355), and secrets-manager integration (JA-8818).

Spec Dependencies

This capability acts as the critical foundational dependency for several other in-flight Omnia features.

Upstream Dependencies (What this feature needs)

• BR-19442 (Diskful Boot Option with OpenCHAMI): Relies on the early-boot provisioning and Golden Image pipelines established by the OpenCHAMI integration to inject files during Day 1 provisioning.

Downstream Dependencies (What this feature unblocks)

1. JA-8818 (External Secrets Management): The node cannot verify the external secrets manager's TLS certificate without the Root CA distributed by this feature.
2. JA-17355 (Shared mTLS telemetry authentication): Compute nodes require the Enterprise CA to trust the approved mTLS telemetry endpoint.
3. JA-20781 (NVIDIA NGC Operators & Containers): Kubernetes worker nodes require the Enterprise CA to pull images from internal private registries securely.

---

Documentation

• Outline CA certificate path structures in security deployment guide.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Race Conditions	Technical	High	Medium	If kubelet or the vmagent starts before the provisioning mechanism finishes updating the CA trust store, connections will fail. Mitigate by enforcing strict systemd dependencies (After=cloud-init.target).

---

Acceptance Criteria

• AC-001: An administrator can place a custom .pem CA certificate on the OIM node and define its path in the Omnia inventory. [Req 1]
• AC-002: Bare-metal nodes provisioned by OpenCHAMI successfully boot and automatically have the CA certificate present in their OS trust store. [Req 1]
• AC-003: Services running on the provisioned node can successfully negotiate a TLS connection with an internal server using a certificate signed by the custom CA, without utilizing insecure skip-verify flags. [Req 1]