[FEATURE] InferenceDeployment: one resource that owns a Model and an InferenceService with model-derived defaults

Open
#2,132 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
ai, infrastructure

Research direction

Start with catalog/catalog.yaml and the llmkube deploy implementation to understand how catalog defaults currently produce a Model and an InferenceService. Then inspect the existing CRD and controller entry points for those resources. The work is complete when an InferenceDeployment reconciles and owns both children, applies model-derived defaults with template overrides, and reports their status; the issue leaves ownership details and rollout behavior open.

Written by the indexing model from the issue text.

Description

component/api component/controller kind/feature priority/medium size/large

Feature Description

An InferenceDeployment CRD: one root resource that owns and reconciles a Model and an InferenceService, filling in opinionated defaults from the selected model. It works like a Kubernetes Deployment, which owns ReplicaSets: users declare the model and how to serve it, and the controller creates the pair, keeps it in sync, and garbage-collects it with the parent.

Problem Statement

Serving one model today takes two resources that have to agree with each other. In practice, most of an InferenceService is derived from the Model and the runtime:

  • Paired by hand: spec.modelRef must name the Model, names are usually kept identical, and deleting one leaves the other behind.
  • Model-specific runtime settings live on the InferenceService, and every user rediscovers them. From one vLLM deployment of a 190 GiB NVFP4 checkpoint:
    • --served-model-name so /v1/models isn't the cache path (#1988);
    • tool-call and reasoning parsers for the model family (#2119);
    • a startup budget sized to the checkpoint (#2118);
    • a memory-backed /dev/shm for tensor parallelism (#2111, now automatic);
    • compile-cache persistence (#2117);
    • tensor-parallel size and GPU count from the quantized size.
  • The catalog only works imperatively. llmkube deploy <catalog-id> already creates the Model + InferenceService pair with defaults from catalog/catalog.yaml. But it's a one-shot CLI action, so it doesn't fit GitOps, and its defaults never update after the resources exist.

Proposed Solution

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceDeployment
metadata:
  name: glm-5-3-flash
spec:
  model:
    # either a catalog entry...
    catalogRef: glm-5.3-flash-nvfp4
    # ...or an explicit source, as on Model today
    # source: nvidia/GLM-5.3-Flash-NVFP4
    # files: ["config.json", "*.json", "*.jinja", "*.safetensors"]
  replicas: 2
  # Anything set here overrides the derived defaults; same schema as
  # InferenceService.spec, like a Deployment's pod template.
  template:
    spec:
      resources:
        memory: 760Gi
      affinity: {}
status:
  modelRef: glm-5-3-flash
  inferenceServiceRef: glm-5-3-flash
  phase: Ready
  conditions: []

Controller behaviour:

  • Creates and owns a Model and an InferenceService (owner references, so deleting the parent cleans up both). It reconciles drift back to the desired state, as a Deployment does with its ReplicaSet.
  • Resolves defaults from the catalog entry, or from the model's metadata when there's no catalog entry: runtime, GPU count and tensor-parallel size, context length, parsers, served name, startup budget and cache settings. It merges spec.template over them, so user values always win.
  • Surfaces child status on the parent: model staged, service ready, endpoint.
  • Optionally keeps rollout history, as Deployments do. A change to the model source could create a new Model and InferenceService revision and switch traffic when it's Ready, instead of mutating the live pair. That could be phase 2.

Alternatives Considered

  • Keep the CLI as the composition layer: works for imperative use, but GitOps users end up hand-maintaining both resources and copying catalog defaults.
  • Inline the Model into InferenceService (spec.model instead of spec.modelRef): simpler, but it loses the ability to share one staged Model across several InferenceServices, and changes an existing API.
  • Helm or Kustomize templates per model: puts the defaults outside the operator, so they can't react to status, such as staged size, or to model metadata.

Open Questions

  • Ownership of child fields: can users edit the generated InferenceService directly? Server-side apply with the parent as field manager would allow it for fields the parent doesn't set, and correct drift on those it does.
  • Shared Models: allow spec.model.ref to an existing Model that the parent doesn't own, for one-Model-many-services setups.
  • Catalog delivery: keep the catalog embedded in the operator, or add a cluster-scoped catalog resource so platform teams can publish their own vetted entries, such as vLLM / NVFP4 settings for their hardware?
  • Naming: InferenceDeployment versus something like ModelDeployment.

Additional Context

  • Prior art in this repo: catalog/catalog.yaml and llmkube deploy, which already pair Model + InferenceService with catalog defaults.
  • Related: #1988, #2111, #2117, #2118, #2119, #2120. Each is a default this resource could apply once per model instead of once per user.
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.