Workload (Non-service) Kubernetes Cluster Functional Group

The issue names no files, so start by finding where Omnia's playbooks assign nodes to functional groups and where the telemetry and Slurm deployment tasks run. Decide how a Workload group skips those tasks without failing, then confirm that the playbook completes with telemetry disabled. Done means AC-001 to AC-004 hold, with the design settled first, since the issue leaves the scope open.

Workload (Non-service) Kubernetes Cluster Functional Group

---

Summary

AI Infrastructure Administrators and Data Scientists need the ability to provision raw, workload-ready Kubernetes clusters dedicated entirely to user jobs without the overhead of internal Omnia telemetry and Slurm services. Introducing a "Workload Kubernetes" functional group into Omnia deployment provides a unified provisioning stack for users who need K8s at scale but do not fit a managed AI platform profile.

---

Problem Statement

Currently, Omnia's Kubernetes cluster architecture is strictly coupled to running internal service node functions (e.g., telemetry, Slurm controllers), assuming Slurm will handle all actual user workloads. This monolithic architecture prevents customers from provisioning Kubernetes clusters explicitly for containerized user jobs. For AI workloads that require Kubernetes at scale, this limitation forces users toward alternative provisioning stacks. This gap prevents Omnia from providing a complete software stack for large-scale AI deployments.

---

Value Proposition

Customer Value

Customers can seamlessly provision dedicated, telemetry-free Kubernetes clusters strictly for running user AI workloads and jobs. This allows them to use Omnia as a unified provisioning tool for both Slurm-based HPC and K8s-based AI workloads, reducing administrative overhead and eliminating the need to purchase or manage expensive third-party enterprise Kubernetes platforms.

---

User Personas

• AI Infrastructure Admin: Manages the deployment and lifecycle of AI compute resources. Needs the ability to provision raw, workload-ready Kubernetes clusters without internal Omnia telemetry overhead getting in the way of user jobs.
• Data Scientist: Submits containerized AI training and inference jobs directly to a native Kubernetes cluster API.

---

Use Cases

Use Case 1: Provisioning a Workload-Only K8s Cluster

Actor: AI Infrastructure Admin
Context: A new AI research group requires a dedicated Kubernetes cluster for their containerized workloads.
Action: Admin selects the "Workload Kubernetes" functional group during Omnia deployment.
Result: Omnia provisions a clean Kubernetes cluster dedicated to user jobs, omitting internal Slurm controllers and central OIM telemetry agents to maximize available compute.

Use Case 2: K8s Native Job Submission

Actor: Data Scientist
Context: The user has standard K8s manifests (Helm charts, Pod specs) for their AI training jobs.
Action: The user logs into the Workload K8s cluster and submits their jobs directly to the K8s API.
Result: The jobs execute natively on the Workload K8s cluster compute nodes.

---

Scope Definition

Goals (Execution Outcomes)

• Introduce a distinct "Workload" Kubernetes functional group in the Omnia deployment configuration.
• Ensure the Workload K8s cluster does not install or run internal Omnia telemetry services (Prometheus/Grafana) or Slurm controllers by default.
• Enable users to submit jobs directly to K8s login nodes.

Non-Goals (Scope Boundaries)

• Supporting multiple concurrent sub-clusters (Service K8s + Workload K8s) managed by a single OIM in this increment (deferred to long-term roadmap).
• Replacing enterprise turn-key MLOps platforms.

---

Requirements

Requirement 1: Workload Kubernetes Functional Group [Req 1]

The system must provide an explicit functional group definition to deploy a Kubernetes cluster intended exclusively for user workloads.

Relates to: Use Case 1 - Provisioning a Workload-Only K8s Cluster

Scenario: Deploying Workload K8s

• When the admin assigns nodes to the Workload Kubernetes functional group, Omnia provisions a standard K8s control plane and worker nodes.
• omits the deployment of internal Omnia service containers (Slurm, central telemetry)

Scenario: Deployment without Telemetry

• When it reaches the telemetry and monitoring tasks, the system detects the functional group and skips the central telemetry deployment safely without failing.

Requirement 2: Telemetry Decoupling [Req 2]

The system must allow the Workload K8s cluster to operate independently of the central OIM telemetry stack, with the option to forward metrics to dedicated service nodes.

Relates to: Use Case 1 - Provisioning a Workload-Only K8s Cluster

Scenario: Decoupled Metrics

• When the cluster is running user jobs, local node metrics are either collected by the cluster's own dedicated service nodes or omitted entirely based on admin configuration.
• the absence of central telemetry does not cause deployment failures

Scenario: Optional Dedicated Metrics

• When the admin specifies dedicated service nodes for the workload cluster, metrics are routed to those specific nodes instead of the central OIM.

Requirement 3: Native K8s Job Submission [Req 3]

The system must provide accessible K8s control plane endpoints (login nodes) allowing end users to authenticate and submit native K8s workloads.

Relates to: Use Case 2 - K8s Native Job Submission

Scenario: User Job Submission

• When a Data Scientist authenticates to the K8s API via a login node, they can successfully apply K8s workload manifests (e.g., Pods, Deployments) which schedule onto the cluster's compute nodes.

Scenario: Unauthorized Access Attempt

• When an unauthenticated user attempts to hit the K8s API, the request is rejected by standard K8s RBAC policies.

---

Security

• Native Kubernetes RBAC must be configured on the login nodes to restrict user access to appropriate namespaces and prevent privilege escalation on the workload nodes.

---

Performance

Omitting the internal Omnia telemetry stack and Slurm controller workloads from the Workload K8s cluster guarantees 100% of node compute and memory overhead is preserved for AI training and user jobs.

---

Telemetry

This feature must not fail silently. All components must emit concrete health signals to the central Omnia observability stack:

• Logs: All service and execution logs must be formatted as structured JSON and forwarded to the central Vector aggregator.
• Metrics: If the component runs as a daemon/service, it must expose a Prometheus-compatible /metrics scrape endpoint (or utilize an LDMS sampler) tracking uptime, execution success/failure rates, and error counts. Although internal Omnia telemetry services are omitted from the Workload cluster nodes to preserve compute, the K8s control plane metrics (API server, kubelet) must still be scraped and forwarded to the central OIM or a dedicated tenant service node for health monitoring.

---

Documentation

• Architecture Docs: Document the architectural differences between Service K8s and Workload K8s, including guidance on when to deploy each.
• Admin Guide: Instructions on assigning nodes to the Workload Kubernetes functional group.

---

Dependencies

• Omnia Playbooks: Any existing Omnia playbooks that hardcode assumptions about the presence of the telemetry stack or Service K8s must be refactored to treat them as optional modular components.
• Downstream Capabilities: This spec provides the foundational workload Kubernetes cluster that downstream capabilities build upon, such as JA-20781 (NVIDIA NGC Operators and Container Examples), which requires a vanilla workload K8s cluster to deploy NVIDIA operators and NGC container workloads.

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Playbook Failures due to Missing Telemetry	Technical	High	High	Rigorous CI/CD testing of the Omnia Playbooks dependency with all telemetry modules disabled to catch hardcoded assumptions.

---

Acceptance Criteria

• AC-001: An administrator can successfully provision a K8s cluster using the Workload functional group without installing Slurm or central telemetry services. [Req 1]
• AC-002: The Omnia deployment playbook successfully completes even when the central telemetry stack is disabled or omitted. [Req 1]
• AC-003: The Workload K8s cluster can optionally send metrics to its own dedicated service nodes rather than the central OIM. [Req 2]
• AC-004: An end-user can authenticate to the Workload K8s login node and successfully submit native Kubernetes jobs via the API. [Req 3]

---

Scope & Sequencing

Supporting multiple concurrent sub-clusters (e.g., managing both a Service K8s and a Workload K8s cluster from a single centralized OIM) is explicitly deferred to a future roadmap increment.

---

Assumptions / Open Questions

Assumptions

• A1: Customers requiring this functional group have containerized their workloads and do not require Slurm scheduling.

Open Questions

• None.

---