[FEATURE] Runtime-aware model catalog: vLLM and SGLang entries with safetensors sources, parsers and served name

Open
#2,133 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes, yaml
Domain
backend, devtools

Research direction

Read pkg/cli/catalog.go and catalog/catalog.yaml, then trace how llmkube deploy maps catalog entries to Model and InferenceService fields. The issue also points to #2119 for typed parser and extra argument fields. Done means existing entries still parse with the default runtime, runtime-specific fields are validated, the catalog lists runtimes, and end-to-end validated vLLM or SGLang entries are added.

Written by the indexing model from the issue text.

Description

area/model-registry component/cli enhancement kind/feature priority/medium size/medium

Feature Description

Extend the catalog schema (catalog/catalog.yaml, pkg/cli/catalog.go) so an entry can describe a vLLM or SGLang deployment of a safetensors checkpoint, not only a GGUF for llama.cpp. This is step 2 of #2132.

Problem Statement

The catalog has 17 entries, all GGUF for llama.cpp. Its Model struct (pkg/cli/catalog.go:38-51) carries source, quantization, context_size, gpu_layers and CPU/memory resources, with no field for the runtime, a repo source with a file set, parsers, the served model name or a tensor-parallel size. llmkube deploy <catalog-id> takes the runtime only from --runtime (default llamacpp), so a catalog entry cannot say "serve this NVFP4 checkpoint on vLLM with these parsers". Users running safetensors checkpoints on vLLM or SGLang rediscover those settings per deployment (#1988, #2119), and an InferenceDeployment.catalogRef (#2132) has nothing to point at.

Proposed Solution

Additive fields, so every existing entry keeps working (runtime defaults to llamacpp):

some-model-nvfp4:
  name: "..."
  runtime: vllm                  # llamacpp (default) | vllm | sglang
  source: org/Model-NVFP4        # repo ID, as on Model.spec.source
  files: ["*.safetensors", "*.json", "*.jinja"]
  quantization: NVFP4
  context_size: 131072
  gpu:
    count: 4
    tensor_parallel_size: 4
  served_model_name: Model-NVFP4
  parsers:
    tool_call: <parser>          # runtime-specific names
    reasoning: <parser>
  extra_args: []
  validated:                     # where the entry was actually run
    hardware: "..."
    llmkube: "0.10.x"
  • llmkube deploy maps the new fields onto the Model (source, files) and the InferenceService (runtime, vllmConfig.tensorParallelSize or the SGLang equivalent, served name and parsers; through extraArgs until #2119's typed fields land).
  • llmkube catalog list shows the runtime.
  • A test parses every entry and rejects runtime-specific fields on the wrong runtime (for example gpu_layers on vLLM, parsers on llama.cpp).
  • Seed it with entries we have run end to end, each with its validated block, rather than settings copied from model cards.

Alternatives Considered

  • A separate catalog file per runtime: duplicates the shared fields and splits catalog list.
  • A cluster-scoped catalog resource (#2132, open question): a later delivery mechanism. This schema is what such a resource would carry, so it comes first either way.

Additional Context

  • Related issues: #2132, #2119, #1988, #2118
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.