[FEATURE] ModelRouter LoadAware: weighted scoring and approximate prefix affinity

Open
#2,230 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
ai, backend

Research direction

Read the LoadAware picker and the linked #2172 and #2174 context first; the issue does not name source files or tests. Check the #2174 benchmark on 0.10.11 and whether production traffic has long prompts without session headers before deciding whether either follow-up is warranted. Done means implementing only the justified follow-up, with configurable scoring defaults that preserve current placement or prefix affinity for headerless requests, and tests showing the expected behavior.

Written by the indexing model from the issue text.

Description

area/performance area/routing enhancement kind/feature priority/medium size/large status/blocked

Feature Description

Two LoadAware follow-ups from #2172 that haven't been built: weighted scoring in place of the picker's fixed order (#2172's PR 3), and approximate prefix affinity for traffic that carries no session header (#2172's PR 4). #2172 proposed them alongside session bindings (#2173, shipped in 0.10.8) and token-weighted load (#2174). This issue keeps them tracked once #2172 closes.

Problem Statement

  • Fixed order instead of weights. The picker orders candidates by a fixed rule: the bound pod unless pressure moves the session, then the lowest load score. Users can't tune how much affinity is worth against load. llm-d's max-score picker sums weighted 0-1 scores per pod, so affinity wins at light load and load overrides it only when a pod is clearly worse on several signals at once.
  • No affinity without a session header. Placement keys on the session header (sessionAffinity.headers). Requests without one get load-based placement only, so a client that resends a long shared prefix without a header loses its cache on every move. In #2172's runs, llm-d's approximate prefix matching reached an 86% prefix hit rate with no header.

Proposed Solution

As #2172 describes:

  • Weighted scoring (#2172 PR 3): split the picker into scorer functions (affinity, prefill tokens, in-flight requests, waiting requests, KV use), each returning 0-1, with weights under balancing. The default weights must reproduce today's placement.
  • Approximate prefix affinity (#2172 PR 4): hash the chat messages in fixed-size chained blocks, keep a per-endpoint LRU sized from the pod's KV capacity, and use the longest-match pod as the affinity target when there's no session header. When a header is present, the match confirms the bound pod. It would also let #2174's estimate use the real matched length instead of prompt growth.

Start only on evidence. #2174's bench on 0.10.11 decides whether either is needed. With session bindings and the small-pool guard on 0.10.8, Qwen measured 2.60 s TTFT p95 against llm-d's 2.11 s, and GLM matched its old sticky-hash setup with 0 spills (#2172). If token-weighted load closes the Qwen gap, PR 3 has no measured benefit. PR 4 matters only if production traffic includes long prompts without a session header.

Alternatives Considered

  • llm-d's precise prefix mode, driven by vLLM KV events. #2172 couldn't get it working, and on hybrid-attention models it reports coarse blocks (about 832 tokens), so approximate matching is the better start.
  • Reusing llm-d's scheduler. Out of scope: LoadAware is LLMKube's own picker.

Additional Context

  • From #2172 (PR 3 and PR 4 sections), which links the llm-d v1.0.0 sources for the max-score picker, the approximate prefix producer and the prefix scorer.
  • #2021, where these were first parked, was closed as superseded by #2173.
  • Related: #2174 (token-weighted load).

Priority

  • Critical - Blocking my use case
  • High - Would significantly improve my workflow
  • Medium - Nice to have
  • Low - Minor improvement

Willingness to Contribute

  • Yes, I can submit a PR
  • Yes, I can help test
  • No, but I can provide feedback
  • No, just requesting
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.