[FEATURE] Runtime-aware model catalog: vLLM and SGLang entries with safetensors sources, parsers and served name
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Read pkg/cli/catalog.go and catalog/catalog.yaml, then trace how llmkube deploy maps catalog entries to Model and InferenceService fields. The issue also points to #2119 for typed parser and extra argument fields. Done means existing entries still parse with the default runtime, runtime-specific fields are validated, the catalog lists runtimes, and end-to-end validated vLLM or SGLang entries are added.
Written by the indexing model from the issue text.
Description
Feature Description
Extend the catalog schema (catalog/catalog.yaml, pkg/cli/catalog.go) so an entry can describe a vLLM or SGLang deployment of a safetensors checkpoint, not only a GGUF for llama.cpp. This is step 2 of #2132.
Problem Statement
The catalog has 17 entries, all GGUF for llama.cpp. Its Model struct (pkg/cli/catalog.go:38-51) carries source, quantization, context_size, gpu_layers and CPU/memory resources, with no field for the runtime, a repo source with a file set, parsers, the served model name or a tensor-parallel size. llmkube deploy <catalog-id> takes the runtime only from --runtime (default llamacpp), so a catalog entry cannot say "serve this NVFP4 checkpoint on vLLM with these parsers". Users running safetensors checkpoints on vLLM or SGLang rediscover those settings per deployment (#1988, #2119), and an InferenceDeployment.catalogRef (#2132) has nothing to point at.
Proposed Solution
Additive fields, so every existing entry keeps working (runtime defaults to llamacpp):
some-model-nvfp4:
name: "..."
runtime: vllm # llamacpp (default) | vllm | sglang
source: org/Model-NVFP4 # repo ID, as on Model.spec.source
files: ["*.safetensors", "*.json", "*.jinja"]
quantization: NVFP4
context_size: 131072
gpu:
count: 4
tensor_parallel_size: 4
served_model_name: Model-NVFP4
parsers:
tool_call: <parser> # runtime-specific names
reasoning: <parser>
extra_args: []
validated: # where the entry was actually run
hardware: "..."
llmkube: "0.10.x"
llmkube deploymaps the new fields onto the Model (source,files) and the InferenceService (runtime,vllmConfig.tensorParallelSizeor the SGLang equivalent, served name and parsers; throughextraArgsuntil #2119's typed fields land).llmkube catalog listshows the runtime.- A test parses every entry and rejects runtime-specific fields on the wrong runtime (for example
gpu_layerson vLLM,parserson llama.cpp). - Seed it with entries we have run end to end, each with its
validatedblock, rather than settings copied from model cards.
Alternatives Considered
- A separate catalog file per runtime: duplicates the shared fields and splits
catalog list. - A cluster-scoped catalog resource (#2132, open question): a later delivery mechanism. This schema is what such a resource would carry, so it comes first either way.
Additional Context
- Related issues: #2132, #2119, #1988, #2118
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day