[FEATURE] Capacity and headroom metrics: per-model limits, peak KV and concurrency, per-pod attribution
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go, kubernetes, prometheus
- Domain
- backend, infrastructure, observability
Research direction
This issue coordinates three pull requests across controller metrics and configuration, router endpoint labels and session counters, and load-scraper histograms; it names internal/router/load_scraper.go and endpoint_watch.go. Start by reading the linked PRs #2574, #2575, and #2576 and their dependencies. Done means all four feature parts are implemented, with tests and metrics matching the described labels, limits, buckets, and lifecycle behavior.
Written by the indexing model from the issue text.
Description
Feature Description
A set of controller and router-proxy metrics that answer "how close is each model to its limits at peak, and on which pod". The set is complete for that question: it covers what the engines and dcgm-exporter don't already export, and nothing they do.
| # | Component | Addition |
|---|---|---|
| 1 | Controller, API | Allow spec.parallelSlots above 64 for vLLM and vllm-swift. Export each InferenceService's limits and config as gauges, and pass the concurrency limit into the ModelRouter's compiled backend config. |
| 2 | Router proxy | Time-weighted histograms of each pod's KV use, slot utilisation, running and waiting requests, from the samples the load scraper already takes once a second, plus a counter of the samples it missed. |
| 3 | Router proxy | target_pod and target_node labels on every per-endpoint metric. |
| 4 | Router proxy | A counter of session bindings created. |
The four parts land as three PRs (1; 3 + 4; 2). PR #2576 and PR #2574 are independent. PR #2575 is stacked on PR #2574, because its histograms carry PR #2574's target_pod and target_node labels. PR #2576 and PR #2575 declare the same ConcurrencyLimit field with identical text, so whichever merges second resolves a trivial conflict.
Problem Statement
As an operator or capacity planner running LLMKube on a fixed GPU fleet, I want to see, per model and per pod, how close each one runs to its concurrency and KV limits at peak, so that I know how much more load fits before I add hardware and which model to move first.
Today:
- The limits aren't exported. Above 64, vLLM's
--max-num-seqscan only be set inextraArgs, becauseparallelSlots(which already renders as--max-num-seqs) hasMaximum=64, a cap sized for llama.cpp slots. Neither the controller nor vLLM exports the value, so every "% of capacity" chart hard-codes it and goes stale on the next change. - Peaks are invisible.
vllm:kv_cache_usage_perc,vllm:num_requests_running/_waitingandllmkube_router_endpoint_loadare gauges. At a 60 s scrape most peaks are missed; at the 300 s intervals some managed backends use for non-production clusters, almost all are. There is no way to ask "p99 KV this week" or "share of the day at or above the 0.9 KV threshold". - Pods can't be identified. Per-endpoint router series carry
endpoint="http://<pod IP>:<port>". Pod IPs change on restart, and the series can't be joined with dcgm-exporter (pod) or the engine's own metrics. - Config changes can't be compared. There is nothing to mark when a limit or context length changed, so "did raising the limit from 104 to 128 help?" means digging through git history.
The router proxy already reads running, waiting and kv_usage from every pod of a LoadAware backend once a second (internal/router/load_scraper.go) and keeps only the latest sample. Recording those samples adds no new scraping.
Proposed Solution
1. Controller: limits and config (PR #2576)
- Raise
parallelSlots'Maximumto 1024. A CEL rule keeps 64 for every runtime except vLLM and vllm-swift. Update the field's doc comment, which still describes only llama.cpp. ExistingextraArgsusers are unaffected:appendMaxNumSeqsArgsalready errors when both are set. - New gauges, labelled
namespace, inferenceservice, set during reconcile besidellmkube_inferenceservice_replicasand removed with it:llmkube_inferenceservice_concurrency_limit: per replica. Read from the flag the controller renders,extraArgsfirst and then the spec field, the same ruleservedContextLengthalready follows: vLLM--max-num-seqs/parallelSlots, SGLang--max-running-requests/sglangConfig.maxRunningRequests, llama.cpp--parallel/parallelSlots. Flag names are read as vLLM reads them (--max_num_seqsis--max-num-seqs), and vLLM's limit is multiplied by--data-parallel-size, since it applies per rank. Underspec.commandonlyspec.argsandextraArgscount. On the metal-agent the engine's own flag is read perspec.runtime. No series when the runtime picks its own, or when the controller can't know the limit (llama.cpp-np, vLLM ranks placed outside the pod, an agent engine with no slot flag, or an unset agent runtime).llmkube_inferenceservice_context_length: from the resolved context size. No series when unknown.llmkube_inferenceservice_gpus_per_replicallmkube_inferenceservice_config_info{generation, runtime, image}= 1. One series per live spec generation; the old one is deleted, so the set doesn't grow over time. An image change also starts a new series.imageis empty for services the metal-agent runs, whose engine isn't a container image.
- Compile the concurrency limit into the ModelRouter backend config (beside
modelInfo), for endpoint backends only, so the router can compute slot utilisation.
Operational impact. The router proxy reads its config once at startup, and the controller restarts it when the config hash changes. So the first reconcile after upgrading rolls every ModelRouter with an InferenceService-backed endpoint backend, and so does any later change to a limit. Session bindings are held in memory and are lost on the roll; sessions are re-placed and may miss their prefix cache once. Routers without such backends keep their hash. In practice the upgrade roll coincides with the one a release already causes when the chart moves the router-proxy image to the new version, and a limit change usually restarts the engine pods too.
2. Router: pressure histograms (PR #2575, on #2574)
Observed on every successful load-scraper sample, labelled router, backend, endpoint, pod, node:
| Metric | Buckets |
|---|---|
llmkube_router_endpoint_kv_usage |
0.1, 0.25, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.98, 0.99, 1 |
llmkube_router_endpoint_slot_utilization (running ÷ limit; only when the limit is known) |
same as above |
llmkube_router_endpoint_running_requests |
0, 1, 2, 4, 8, 16, 32, 64, 96, 128, 192, 256, 512 |
llmkube_router_endpoint_waiting_requests |
same as running |
The utilisation histogram is the main one. A count histogram is coarse exactly where limits sit (64 → 128), while a 0–1 histogram packed toward 1 resolves every model's peak without tuning. The raw running count stays because planners size in concurrent agents, not percentages.
llmkube_router_endpoint_load_scrape_failures_total counts the samples the histograms miss. A scrape that times out (500 ms) records nothing, and an engine is slowest to answer /metrics at peak load, so that is where missed samples cluster. A failed sample never reaches the histogram's _count, so the share missing is the failure rate over the failure rate plus the rate of llmkube_router_endpoint_running_requests_count. Check it before trusting the top percentiles.
Series are removed when an endpoint leaves, or when its URL comes to point at a different pod. Cost: about 65 series per endpoint per router replica. With several router replicas each replica records every pod, so counts multiply by the number of replicas; quantiles and ratios don't change.
# p99 slot utilisation per model over a week
histogram_quantile(0.99, sum by (backend, le) (rate(llmkube_router_endpoint_slot_utilization_bucket[7d])))
# share of time a model's pods spent at or above the KV-full threshold
1 - sum by (backend) (rate(llmkube_router_endpoint_kv_usage_bucket{le="0.85"}[1d]))
/ sum by (backend) (rate(llmkube_router_endpoint_kv_usage_count[1d]))
3. Router: target_pod and target_node labels (PR #2574)
The EndpointSlices endpoint_watch.go already watches carry each endpoint's TargetRef (the Pod) and NodeName; until now ReadyEndpointURLs kept only the address. Keep both beside the URL and add them as labels on llmkube_router_endpoint_load, _metrics_fresh, _prefill_tokens and llmkube_router_prefill_tokens_estimated_total. Empty for static Address backends. They are not named pod and node, which a scrape sets to the router-proxy pod's own identity; under the chart's PodMonitor the metric's labels would be stored as exported_pod and exported_node. A join with dcgm-exporter copies target_pod into pod with label_replace. When a reused pod IP comes to point at a different pod, the old pod's series are dropped. The number of series doesn't change.
4. Router: llmkube_router_session_bindings_created_total{router, backend} (PR #2574)
Counts session keys a router-proxy replica binds while holding no binding for them: a key's first request, or its first after the binding expired. Moving a bound session to another pod is already a spill or migrated placement and doesn't count. Bindings live in each replica's memory, so a proxy restart, a session hashed to another replica, or a session idle past idleTimeout binds again. Its rate is therefore an upper bound on sessions started while the binding table has room, not an exact count. A session that starts while the table is full falls back unbound and is not counted. It is still the closest signal the router has for how many sessions each user starts.
Alternatives Considered
- A shorter scrape interval: costs more on every metric and still misses peaks shorter than the interval.
- Recording rules (
max_over_time) in the chart's PrometheusRule: only works where a Prometheus evaluates them; backends that receive data by remote write don't. - A histogram of in-flight requests at each request's arrival: weighted by request, not time. It doesn't cover KV, and it overstates busy periods.
- A typed
vllmConfig.maxNumSeqs: a duplicate ofparallelSlots, which the vLLM path already maps. - Reading the limit from the spec field only: deployments above 64 have had to put it in
extraArgs, so they would get no limit until they migrate. The controller already readsextraArgsfirst for the context length. - Computing utilisation in queries from the controller's gauge instead of passing the limit to the router: avoids the router restarts, but a running-count histogram is too coarse at the limit to give a usable ratio.
- Replacing
endpointwithtarget_pod: breaks existing dashboards and recording rules; adding labels is backward compatible. - Naming the labels
podandnodeand settinghonorLabels: trueon the router PodMonitor: takes the router pod's own identity off every router series.
Additional Context
- Out of scope, already exported elsewhere: tokens, TTFT, inter-token latency, queue time, preemptions and prefix-cache hits per model (vLLM); GPU busy and memory per pod (dcgm-exporter, whose
--kubernetes-enable-pod-labels, or gpu-operator'sdcgmExporter.enablePodLabels, adds theinference.llmkube.dev/*pod labels); request outcomes per backend (llmkube_router_requests_total). - Histograms (2) cover only backends behind a LoadAware router, the only ones whose engine load the router reads.
- Related: #1145 and #1146 (per-request engine metrics for SLOs).
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day