[FEATURE] ModelRouter LoadAware: weighted scoring and approximate prefix affinity
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Read the LoadAware picker and the linked #2172 and #2174 context first; the issue does not name source files or tests. Check the #2174 benchmark on 0.10.11 and whether production traffic has long prompts without session headers before deciding whether either follow-up is warranted. Done means implementing only the justified follow-up, with configurable scoring defaults that preserve current placement or prefix affinity for headerless requests, and tests showing the expected behavior.
Written by the indexing model from the issue text.
Description
Feature Description
Two LoadAware follow-ups from #2172 that haven't been built: weighted scoring in place of the picker's fixed order (#2172's PR 3), and approximate prefix affinity for traffic that carries no session header (#2172's PR 4). #2172 proposed them alongside session bindings (#2173, shipped in 0.10.8) and token-weighted load (#2174). This issue keeps them tracked once #2172 closes.
Problem Statement
- Fixed order instead of weights. The picker orders candidates by a fixed rule: the bound pod unless pressure moves the session, then the lowest load score. Users can't tune how much affinity is worth against load. llm-d's max-score picker sums weighted 0-1 scores per pod, so affinity wins at light load and load overrides it only when a pod is clearly worse on several signals at once.
- No affinity without a session header. Placement keys on the session header (
sessionAffinity.headers). Requests without one get load-based placement only, so a client that resends a long shared prefix without a header loses its cache on every move. In #2172's runs, llm-d's approximate prefix matching reached an 86% prefix hit rate with no header.
Proposed Solution
As #2172 describes:
- Weighted scoring (#2172 PR 3): split the picker into scorer functions (affinity, prefill tokens, in-flight requests, waiting requests, KV use), each returning 0-1, with weights under
balancing. The default weights must reproduce today's placement. - Approximate prefix affinity (#2172 PR 4): hash the chat messages in fixed-size chained blocks, keep a per-endpoint LRU sized from the pod's KV capacity, and use the longest-match pod as the affinity target when there's no session header. When a header is present, the match confirms the bound pod. It would also let #2174's estimate use the real matched length instead of prompt growth.
Start only on evidence. #2174's bench on 0.10.11 decides whether either is needed. With session bindings and the small-pool guard on 0.10.8, Qwen measured 2.60 s TTFT p95 against llm-d's 2.11 s, and GLM matched its old sticky-hash setup with 0 spills (#2172). If token-weighted load closes the Qwen gap, PR 3 has no measured benefit. PR 4 matters only if production traffic includes long prompts without a session header.
Alternatives Considered
- llm-d's precise prefix mode, driven by vLLM KV events. #2172 couldn't get it working, and on hybrid-attention models it reports coarse blocks (about 832 tokens), so approximate matching is the better start.
- Reusing llm-d's scheduler. Out of scope: LoadAware is LLMKube's own picker.
Additional Context
- From #2172 (PR 3 and PR 4 sections), which links the llm-d v1.0.0 sources for the max-score picker, the approximate prefix producer and the prefix scorer.
- #2021, where these were first parked, was closed as superseded by #2173.
- Related: #2174 (token-weighted load).
Priority
- Critical - Blocking my use case
- High - Would significantly improve my workflow
- Medium - Nice to have
- Low - Minor improvement
Willingness to Contribute
- Yes, I can submit a PR
- Yes, I can help test
- No, but I can provide feedback
- No, just requesting
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day