Forward the engine's LoRA, extra-key, cache-salt and group identity to the router in KV events

Aperta
  • Qualcosa non va?
#2,089 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
backend, distributed-systems

Direzione di ricerca

Start with sim/llmd/kvevents.go and trace how the publisher builds embed.BlockStored from engine stored runs; compare its fields with core/vllm/engine.go, which records the adapter. Check the blis-substrate KVBatch.Engine contract dependency and llm-d's event adapters before estimating the engine-specific behavior. Done means the requested metadata and clear events are forwarded, with tests for adapter and non-adapter routing plus event decoding parity.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What problem does this solve?

The router's prefix index cannot tell a LoRA run from a base-model run, because inference-sim does not forward the LoRA identity of a stored block to it.

blis-vllm's 0.32.0 engine records the adapter on every stored run: core/vllm/engine.go sets sr.LoRAName = r.Adapter and the LoRA key in Keys.Extra. inference-sim's KV-event publisher, sim/llmd/kvevents.go, builds each embed.BlockStored from the hash, parent, tokens and tier only. It never sets LoRAID, LoRAName, ExtraKeys, CacheSalt, GroupIdx, SpecKind or SlidingWindow; a search for those names under sim/llmd finds nothing. The router fork (llm-d-router pull request 5) now carries all of them into llm-d's events, as llm-d's own adapters decode them, but nothing produces them.

Two consequences:

  • A hosted router with the precise prefix-cache scorer indexes LoRA runs under the base model's keys, so a LoRA request can score a base-model endpoint as a prefix hit. Scores, and the endpoint chosen, differ from llm-d's.
  • Once the publisher does forward the name, it must also say which engine it is, or a LoRAName beside a LoRAID is refused by the router's conversion. That needs KVBatch.Engine in blis-substrate's contracts/embed (a blis-substrate issue tracks it).

What would you like to see?

  1. The publisher forwards the LoRA id and name, the extra keys, the cache salt, the group index, the cache-spec kind and the sliding-window size from the engine's stored runs to embed.BlockStored, and AllBlocksCleared where the engine clears.
  2. A test that a LoRA run with precise-prefix routing is routed and not refused, and that its events carry the adapter, with the closest run without an adapter producing events that do not (SIM-9).
  3. A test that compares the events the publisher builds with what llm-d's own adapters decode, so the forwarded fields are held to llm-d's code (X4).

Depends on the KVBatch.Engine contract change (blis-substrate) for the engine-specific rule. Milestone: M2.

Raised while reviewing llm-d-router pull request 5.

Lingua principale
Go
Stelle
40
Fork
28
Merge medio
1g 2h
PR unite (30g)
31

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di inference-sim/inference-sim

Tutte le issue di inference-sim/inference-sim

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.