[FEATURE] InferenceDeployment: one resource that owns a Model and an InferenceService with model-derived defaults
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- ai, infrastructure
Research direction
Start with catalog/catalog.yaml and the llmkube deploy implementation to understand how catalog defaults currently produce a Model and an InferenceService. Then inspect the existing CRD and controller entry points for those resources. The work is complete when an InferenceDeployment reconciles and owns both children, applies model-derived defaults with template overrides, and reports their status; the issue leaves ownership details and rollout behavior open.
Written by the indexing model from the issue text.
Description
Feature Description
An InferenceDeployment CRD: one root resource that owns and reconciles a Model and an InferenceService, filling in opinionated defaults from the selected model. It works like a Kubernetes Deployment, which owns ReplicaSets: users declare the model and how to serve it, and the controller creates the pair, keeps it in sync, and garbage-collects it with the parent.
Problem Statement
Serving one model today takes two resources that have to agree with each other. In practice, most of an InferenceService is derived from the Model and the runtime:
- Paired by hand:
spec.modelRefmust name the Model, names are usually kept identical, and deleting one leaves the other behind. - Model-specific runtime settings live on the InferenceService, and every user rediscovers them. From one vLLM deployment of a 190 GiB NVFP4 checkpoint:
--served-model-nameso/v1/modelsisn't the cache path (#1988);- tool-call and reasoning parsers for the model family (#2119);
- a startup budget sized to the checkpoint (#2118);
- a memory-backed
/dev/shmfor tensor parallelism (#2111, now automatic); - compile-cache persistence (#2117);
- tensor-parallel size and GPU count from the quantized size.
- The catalog only works imperatively.
llmkube deploy <catalog-id>already creates the Model + InferenceService pair with defaults fromcatalog/catalog.yaml. But it's a one-shot CLI action, so it doesn't fit GitOps, and its defaults never update after the resources exist.
Proposed Solution
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceDeployment
metadata:
name: glm-5-3-flash
spec:
model:
# either a catalog entry...
catalogRef: glm-5.3-flash-nvfp4
# ...or an explicit source, as on Model today
# source: nvidia/GLM-5.3-Flash-NVFP4
# files: ["config.json", "*.json", "*.jinja", "*.safetensors"]
replicas: 2
# Anything set here overrides the derived defaults; same schema as
# InferenceService.spec, like a Deployment's pod template.
template:
spec:
resources:
memory: 760Gi
affinity: {}
status:
modelRef: glm-5-3-flash
inferenceServiceRef: glm-5-3-flash
phase: Ready
conditions: []
Controller behaviour:
- Creates and owns a
Modeland anInferenceService(owner references, so deleting the parent cleans up both). It reconciles drift back to the desired state, as a Deployment does with its ReplicaSet. - Resolves defaults from the catalog entry, or from the model's metadata when there's no catalog entry: runtime, GPU count and tensor-parallel size, context length, parsers, served name, startup budget and cache settings. It merges
spec.templateover them, so user values always win. - Surfaces child status on the parent: model staged, service ready, endpoint.
- Optionally keeps rollout history, as Deployments do. A change to the model source could create a new Model and InferenceService revision and switch traffic when it's Ready, instead of mutating the live pair. That could be phase 2.
Alternatives Considered
- Keep the CLI as the composition layer: works for imperative use, but GitOps users end up hand-maintaining both resources and copying catalog defaults.
- Inline the Model into InferenceService (
spec.modelinstead ofspec.modelRef): simpler, but it loses the ability to share one staged Model across several InferenceServices, and changes an existing API. - Helm or Kustomize templates per model: puts the defaults outside the operator, so they can't react to status, such as staged size, or to model metadata.
Open Questions
- Ownership of child fields: can users edit the generated InferenceService directly? Server-side apply with the parent as field manager would allow it for fields the parent doesn't set, and correct drift on those it does.
- Shared Models: allow
spec.model.refto an existing Model that the parent doesn't own, for one-Model-many-services setups. - Catalog delivery: keep the catalog embedded in the operator, or add a cluster-scoped catalog resource so platform teams can publish their own vetted entries, such as vLLM / NVFP4 settings for their hardware?
- Naming:
InferenceDeploymentversus something likeModelDeployment.
Additional Context
- Prior art in this repo:
catalog/catalog.yamlandllmkube deploy, which already pair Model + InferenceService with catalog defaults. - Related: #1988, #2111, #2117, #2118, #2119, #2120. Each is a default this resource could apply once per model instead of once per user.
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day