[FEATURE] Capacity and headroom metrics: per-model limits, peak KV and concurrency, per-pod attribution

Open
#2,573 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes, prometheus

Research direction

This issue coordinates three pull requests across controller metrics and configuration, router endpoint labels and session counters, and load-scraper histograms; it names internal/router/load_scraper.go and endpoint_watch.go. Start by reading the linked PRs #2574, #2575, and #2576 and their dependencies. Done means all four feature parts are implemented, with tests and metrics matching the described labels, limits, buckets, and lifecycle behavior.

Written by the indexing model from the issue text.

Description

Feature Description

A set of controller and router-proxy metrics that answer "how close is each model to its limits at peak, and on which pod". The set is complete for that question: it covers what the engines and dcgm-exporter don't already export, and nothing they do.

# Component Addition
1 Controller, API Allow spec.parallelSlots above 64 for vLLM and vllm-swift. Export each InferenceService's limits and config as gauges, and pass the concurrency limit into the ModelRouter's compiled backend config.
2 Router proxy Time-weighted histograms of each pod's KV use, slot utilisation, running and waiting requests, from the samples the load scraper already takes once a second, plus a counter of the samples it missed.
3 Router proxy target_pod and target_node labels on every per-endpoint metric.
4 Router proxy A counter of session bindings created.

The four parts land as three PRs (1; 3 + 4; 2). PR #2576 and PR #2574 are independent. PR #2575 is stacked on PR #2574, because its histograms carry PR #2574's target_pod and target_node labels. PR #2576 and PR #2575 declare the same ConcurrencyLimit field with identical text, so whichever merges second resolves a trivial conflict.

Problem Statement

As an operator or capacity planner running LLMKube on a fixed GPU fleet, I want to see, per model and per pod, how close each one runs to its concurrency and KV limits at peak, so that I know how much more load fits before I add hardware and which model to move first.

Today:

  • The limits aren't exported. Above 64, vLLM's --max-num-seqs can only be set in extraArgs, because parallelSlots (which already renders as --max-num-seqs) has Maximum=64, a cap sized for llama.cpp slots. Neither the controller nor vLLM exports the value, so every "% of capacity" chart hard-codes it and goes stale on the next change.
  • Peaks are invisible. vllm:kv_cache_usage_perc, vllm:num_requests_running / _waiting and llmkube_router_endpoint_load are gauges. At a 60 s scrape most peaks are missed; at the 300 s intervals some managed backends use for non-production clusters, almost all are. There is no way to ask "p99 KV this week" or "share of the day at or above the 0.9 KV threshold".
  • Pods can't be identified. Per-endpoint router series carry endpoint="http://<pod IP>:<port>". Pod IPs change on restart, and the series can't be joined with dcgm-exporter (pod) or the engine's own metrics.
  • Config changes can't be compared. There is nothing to mark when a limit or context length changed, so "did raising the limit from 104 to 128 help?" means digging through git history.

The router proxy already reads running, waiting and kv_usage from every pod of a LoadAware backend once a second (internal/router/load_scraper.go) and keeps only the latest sample. Recording those samples adds no new scraping.

Proposed Solution

1. Controller: limits and config (PR #2576)
  • Raise parallelSlots' Maximum to 1024. A CEL rule keeps 64 for every runtime except vLLM and vllm-swift. Update the field's doc comment, which still describes only llama.cpp. Existing extraArgs users are unaffected: appendMaxNumSeqsArgs already errors when both are set.
  • New gauges, labelled namespace, inferenceservice, set during reconcile beside llmkube_inferenceservice_replicas and removed with it:
    • llmkube_inferenceservice_concurrency_limit: per replica. Read from the flag the controller renders, extraArgs first and then the spec field, the same rule servedContextLength already follows: vLLM --max-num-seqs / parallelSlots, SGLang --max-running-requests / sglangConfig.maxRunningRequests, llama.cpp --parallel / parallelSlots. Flag names are read as vLLM reads them (--max_num_seqs is --max-num-seqs), and vLLM's limit is multiplied by --data-parallel-size, since it applies per rank. Under spec.command only spec.args and extraArgs count. On the metal-agent the engine's own flag is read per spec.runtime. No series when the runtime picks its own, or when the controller can't know the limit (llama.cpp -np, vLLM ranks placed outside the pod, an agent engine with no slot flag, or an unset agent runtime).
    • llmkube_inferenceservice_context_length: from the resolved context size. No series when unknown.
    • llmkube_inferenceservice_gpus_per_replica
    • llmkube_inferenceservice_config_info{generation, runtime, image} = 1. One series per live spec generation; the old one is deleted, so the set doesn't grow over time. An image change also starts a new series. image is empty for services the metal-agent runs, whose engine isn't a container image.
  • Compile the concurrency limit into the ModelRouter backend config (beside modelInfo), for endpoint backends only, so the router can compute slot utilisation.

Operational impact. The router proxy reads its config once at startup, and the controller restarts it when the config hash changes. So the first reconcile after upgrading rolls every ModelRouter with an InferenceService-backed endpoint backend, and so does any later change to a limit. Session bindings are held in memory and are lost on the roll; sessions are re-placed and may miss their prefix cache once. Routers without such backends keep their hash. In practice the upgrade roll coincides with the one a release already causes when the chart moves the router-proxy image to the new version, and a limit change usually restarts the engine pods too.

2. Router: pressure histograms (PR #2575, on #2574)

Observed on every successful load-scraper sample, labelled router, backend, endpoint, pod, node:

Metric Buckets
llmkube_router_endpoint_kv_usage 0.1, 0.25, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.98, 0.99, 1
llmkube_router_endpoint_slot_utilization (running ÷ limit; only when the limit is known) same as above
llmkube_router_endpoint_running_requests 0, 1, 2, 4, 8, 16, 32, 64, 96, 128, 192, 256, 512
llmkube_router_endpoint_waiting_requests same as running

The utilisation histogram is the main one. A count histogram is coarse exactly where limits sit (64 → 128), while a 0–1 histogram packed toward 1 resolves every model's peak without tuning. The raw running count stays because planners size in concurrent agents, not percentages.

llmkube_router_endpoint_load_scrape_failures_total counts the samples the histograms miss. A scrape that times out (500 ms) records nothing, and an engine is slowest to answer /metrics at peak load, so that is where missed samples cluster. A failed sample never reaches the histogram's _count, so the share missing is the failure rate over the failure rate plus the rate of llmkube_router_endpoint_running_requests_count. Check it before trusting the top percentiles.

Series are removed when an endpoint leaves, or when its URL comes to point at a different pod. Cost: about 65 series per endpoint per router replica. With several router replicas each replica records every pod, so counts multiply by the number of replicas; quantiles and ratios don't change.

# p99 slot utilisation per model over a week
histogram_quantile(0.99, sum by (backend, le) (rate(llmkube_router_endpoint_slot_utilization_bucket[7d])))

# share of time a model's pods spent at or above the KV-full threshold
1 - sum by (backend) (rate(llmkube_router_endpoint_kv_usage_bucket{le="0.85"}[1d]))
  / sum by (backend) (rate(llmkube_router_endpoint_kv_usage_count[1d]))
3. Router: target_pod and target_node labels (PR #2574)

The EndpointSlices endpoint_watch.go already watches carry each endpoint's TargetRef (the Pod) and NodeName; until now ReadyEndpointURLs kept only the address. Keep both beside the URL and add them as labels on llmkube_router_endpoint_load, _metrics_fresh, _prefill_tokens and llmkube_router_prefill_tokens_estimated_total. Empty for static Address backends. They are not named pod and node, which a scrape sets to the router-proxy pod's own identity; under the chart's PodMonitor the metric's labels would be stored as exported_pod and exported_node. A join with dcgm-exporter copies target_pod into pod with label_replace. When a reused pod IP comes to point at a different pod, the old pod's series are dropped. The number of series doesn't change.

4. Router: llmkube_router_session_bindings_created_total{router, backend} (PR #2574)

Counts session keys a router-proxy replica binds while holding no binding for them: a key's first request, or its first after the binding expired. Moving a bound session to another pod is already a spill or migrated placement and doesn't count. Bindings live in each replica's memory, so a proxy restart, a session hashed to another replica, or a session idle past idleTimeout binds again. Its rate is therefore an upper bound on sessions started while the binding table has room, not an exact count. A session that starts while the table is full falls back unbound and is not counted. It is still the closest signal the router has for how many sessions each user starts.

Alternatives Considered

  • A shorter scrape interval: costs more on every metric and still misses peaks shorter than the interval.
  • Recording rules (max_over_time) in the chart's PrometheusRule: only works where a Prometheus evaluates them; backends that receive data by remote write don't.
  • A histogram of in-flight requests at each request's arrival: weighted by request, not time. It doesn't cover KV, and it overstates busy periods.
  • A typed vllmConfig.maxNumSeqs: a duplicate of parallelSlots, which the vLLM path already maps.
  • Reading the limit from the spec field only: deployments above 64 have had to put it in extraArgs, so they would get no limit until they migrate. The controller already reads extraArgs first for the context length.
  • Computing utilisation in queries from the controller's gauge instead of passing the limit to the router: avoids the router restarts, but a running-count histogram is too coarse at the limit to give a usable ratio.
  • Replacing endpoint with target_pod: breaks existing dashboards and recording rules; adding labels is backward compatible.
  • Naming the labels pod and node and setting honorLabels: true on the router PodMonitor: takes the router pod's own identity off every router series.

Additional Context

  • Out of scope, already exported elsewhere: tokens, TTFT, inter-token latency, queue time, preemptions and prefix-cache hits per model (vLLM); GPU busy and memory per pod (dcgm-exporter, whose --kubernetes-enable-pod-labels, or gpu-operator's dcgmExporter.enablePodLabels, adds the inference.llmkube.dev/* pod labels); request outcomes per backend (llmkube_router_requests_total).
  • Histograms (2) cover only backends behind a LoadAware router, the only ones whose engine load the router reads.
  • Related: #1145 and #1146 (per-request engine metrics for SLOs).
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.