Setup shared mTLS authentication for kafka/victoria data pipeline

This is a requirements spec with no named files, tests or entry points, so there is no code to start from. Requirements 1-3 and acceptance criteria AC-001 to AC-006 describe the mTLS and token-auth ingress boundary, and the listener modes and ports are said to live in restricted documentation. Before any work, the scope needs to be settled with maintainers, starting with the dependencies BR-31940 and BR-18602.

Setup shared mTLS authentication for kafka/victoria data pipeline

Summary

The Omnia telemetry workflow requires strict access controls to prevent unauthorized producers from injecting malicious or malformed data. External ingestion endpoints must enforce authenticated TLS or mTLS, including support for approved TLS-only devices that cannot present client certificates. Internal telemetry transport remains confined to a protected cluster network and follows the deployment's approved security policy.

---

Problem Statement

Communication among telemetry databases, brokers, producers, and agents requires one authentication boundary that prevents unauthorized publication. External producers have mixed capabilities: some support mTLS, while approved legacy devices support only standard TLS combined with another authentication method. The deployment must enforce authentication and encryption at external ingestion endpoints while keeping internal transport within a protected cluster network governed by the approved security policy.

---

Value Proposition

Customer Value

Provides a controlled authentication boundary for telemetry producers while preserving internal cluster performance. Both mTLS-capable and TLS-only devices can connect through a unified gateway according to the configured trust policy.

---

User Personas

• Security / Compliance Admin: Mandates that no unauthenticated entity can inject data into the cluster, requiring a certificate-based barrier of entry.
• HPC Platform Admin: Needs to manage external hardware connections to the telemetry stack without managing complex internal Kubernetes mesh encryption.

---

Use Cases

Use Case 1: mTLS External Producer

Actor: Compute Node Agent (Vector)
Context: An Omnia-managed compute node needs to stream high-frequency telemetry into the central Kafka brokers.
Action: The external Vector agent establishes an mTLS connection with the Service Cluster's Ingress Gateway, presenting its valid client certificate.
Result: The approved ingress service authenticates the agent and forwards the payload through the policy-compliant internal telemetry path.

Use Case 2: TLS-Only Legacy Hardware

Actor: HPC Hardware Appliance (e.g., Switch/Storage)
Context: A physical appliance needs to push metrics to VictoriaMetrics but does not support client-side mTLS certificates.
Action: The appliance establishes a standard TLS connection to the approved ingress service and presents a supported authentication token.
Result: The ingress service validates the token and forwards metrics through the policy-compliant internal telemetry path.

Use Case 3: Policy-Controlled Internal Routing

Actor: System (Automated)
Context: Telemetry services need to communicate within the protected cluster network.
Action: Internal components communicate through cluster-local service discovery according to the deployment's approved security policy.
Result: Internal telemetry remains isolated from external networks and is not exposed through unauthenticated endpoints.

---

Scope Definition

Goals (Execution Outcomes)

• Provide an approved ingress service as the unified authentication boundary for telemetry entering the Service Kubernetes cluster.
• Enforce authenticated mTLS or TLS at external ingestion endpoints.
• Configure internal telemetry transport according to the deployment's approved security policy and protected network boundary.
• Separate externally reachable endpoints from cluster-internal service communication.
• Support approved token-based authentication for agents incapable of mTLS.

Non-Goals (Scope Boundaries)

• This specification does not prescribe a service-mesh implementation for internal telemetry transport.
• Exact listener modes, ports, gateway configuration, and credential procedures remain in restricted operational documentation.

---

Requirements

Requirement 1: Authenticated TLS/mTLS Ingress [Req 1]

The system must provide an approved ingress service that authenticates external telemetry producers and routes accepted traffic through a policy-compliant internal transport.

Relates to: Use Case 1 and 2

Scenario: Authenticated external producer

• When an external agent pushes metrics, the approved ingress service validates its mTLS or TLS authentication before forwarding accepted traffic through the internal telemetry path.

Requirement 2: Fallback Authentication [Req 2]

The ingress gateway must support standard TLS combined with token-based or Basic authentication.

Relates to: Use Case 2

Scenario: Legacy Agent Connection

• When a legacy agent connects to the telemetry ingress, it establishes standard TLS and authenticates using a predefined secure token/password validated by the gateway.

Requirement 3: External and Internal Telemetry Boundary [Req 3]

Telemetry brokers must separate externally reachable ingress from cluster-internal service communication.

Relates to: Use Case 1 and 3

Scenario: Broker boundary

• When traffic routes to telemetry brokers, external producers use authenticated encrypted ingress and cluster-internal communication remains confined to the protected network according to the approved security policy.

---

Usability

The Omnia deployment must automatically generate and distribute the required client certificates to the Omnia-managed compute nodes (e.g., Vector agents) during Day 1 provisioning so that mTLS works out-of-the-box without manual key distribution.

---

Security

• External endpoints must enforce authenticated TLS or mTLS. Internal telemetry transport must remain confined to the protected cluster network and follow the deployment's approved security policy.

---

Telemetry

The telemetry authentication boundary must expose supported metrics for authentication success rates, TLS/mTLS processing, and unauthorized access attempts without publishing internal administrative endpoints.

---

Performance

• Terminating TLS at the ingress gateway rather than individual database pods reduces CPU decryption overhead across the VictoriaMetrics cluster, allowing for denser metrics ingestion.

---

Documentation

• Security Architecture Guide: Document the external authentication boundary, supported producer authentication methods, internal network-isolation requirements, and operator procedures without publishing listener modes, ports, or credential-generation details.

---

Dependencies

Cross-Cutting Platform Dependencies

• CA / Certificate Infrastructure: Depends on BR-31940 (CA cert distribution) to deliver trusted CA certificates to compute nodes. Without the CA, vmagent cannot verify the mTLS endpoint.
• IPv6 Compatibility: Certificate SANs must include IPv6 addresses (BR-18602). Kafka and VictoriaMetrics endpoints must be reachable over IPv6 when configured.

Spec Dependencies

• Omnia 2.2 vmagent workflow: The baseline scalable telemetry architecture must be deployed so the approved ingress service has internal telemetry endpoints to route to.
• Service Kubernetes Cluster: Relies on Omnia's foundational Service Cluster architecture to provide the network isolation boundary.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Certificate Management	Operational	High	Medium	Distributing client certificates to thousands of compute nodes can fail if the Certificate Authority (CA) logic is flawed. Mitigate by utilizing Ansible's native crypto modules or integration with a robust internal secret manager during node provisioning.

---

Acceptance Criteria

• AC-001: An approved ingress service successfully authenticates mTLS and TLS traffic entering the Service Kubernetes cluster. [Req 1]
• AC-002: Internal telemetry communication remains confined to the protected cluster network and follows the approved deployment security policy. [Req 3]
• AC-003: The telemetry stack functions with the supported software edition and selected security boundary. [Req 1]
• AC-004: External producers without client certificates can authenticate through standard TLS combined with an approved token or password mechanism. [Req 2]
• AC-005: Unauthorized external producers lacking valid certificates or tokens are blocked at the authentication boundary. [Req 1]
• AC-006: Compute node telemetry agents (Vector) receive auto-generated client certificates during Omnia provisioning and successfully stream data via mTLS. [Usability, Req 1]

---

Scope & Sequencing

Must be sequenced after the core Service Kubernetes cluster deployment and alongside the telemetry stack deployment.

---

Assumptions / Open Questions

Assumptions

• A1: The deployment security policy defines the permitted internal telemetry transport, and external ingress remains authenticated through TLS or mTLS.

Open Questions

• None.

---