Host prefix cache: ~30 % of hits dropped as capacity_rejected and fully recomputed

Aperta
  • Qualcosa non va?
#255 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend, performance

Direzione di ricerca

Start with the host-cache hit admission path in helios/src/server/runtime.rs and resource checks in helios/src/scheduler.rs. Trace how ensure_resources distinguishes pages awaiting GPU retirement from exhausted live resources, then inspect the related admission and host-cache tests. Done means capacity rejection causes are observable or a retry/partial-restore behavior is implemented and covered by tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Setup
  • Flare 972e28e, built from source with one metrics-only patch (adds a model_name label to /metrics).
  • DeepSeek V4.1 Flash 1017 (gb202 payload package), 4 × RTX PRO 6000 Blackwell (SM120), TP4.
  • --scheduler-policy throughput --mtp-draft-tokens 7 --mtp-adaptive-verification --prefix-cache-memory 60G --prefix-cache-disk <dir> --prefix-cache-disk-limit 800G
  • Traffic: agentic coding sessions (long shared prefixes, prompts up to ~165k tokens), ~21k requests/day.
What happens

When a request's prefix is found in the host cache, roughly 30 % of those hits are thrown away because the restore does not fit in GPU KV pages at that moment. The request is then admitted without the restore and the whole prefix is prefilled again.

Last 24 h, flare_host_cache_lookup_outcomes_total:

reason count
no_match 14023
admitted 4068
capacity_rejected 1724
cost_rejected 73

So 1724 of 5792 usable hits (30 %) end up as full recomputation, often of long prefixes. That is exactly the work the host cache exists to avoid.

Where it comes from

helios/src/server/runtime.rs, admission with a host-cache hit:

let admitted = match scheduler.admit_planned(spec.clone(), plan, Some(inbound)) {
    Ok(admitted) => admitted,
    Err(Error::Admission(AdmissionError::Capacity(_))) => {
        let handle = scheduler.admit(spec)?;
        scheduler.record_host_lookup();
        return Ok((handle, Some(LookupOutcome::CapacityRejected)));
    }
    ...

ensure_resources (helios/src/scheduler.rs) returns Capacity for two different reasons:

  • "request resources are awaiting GPU retirement": enough pages exist but are still fenced.
  • "live request resources are exhausted or awaiting GPU retirement": nothing more can be evicted.

In both cases the hit is dropped right away. The plain admit(spec) fallback then succeeds, so the request was admissible. It just could not take all the restore pages at once.

The rejections don't only happen when the pool is empty. Bucketing the 24 h by the minimum of flare_free_kv_pages in each 5-minute window:

min free KV pages (5 min) capacity_rejected
0–10 495
11–100 332
101–1000 417
> 1000 480

(The gauge is sampled, so short dips may be missed. Still, a good share happens with plenty of free pages, which points at fenced pages awaiting retirement and/or restores larger than the instantaneous free set.)

Suggestions (any of these would help)
  1. Split the outcome: count the two Capacity reasons separately (capacity_pending_retirement vs capacity_exhausted), so operators can tell them apart.
  2. Defer instead of dropping on "awaiting GPU retirement": keep the request queued with its hit and retry admission once the fenced pages retire, rather than falling back to a full prefill.
  3. Partial restore: if only part of the hit fits, restore the longest prefix that fits and prefill the rest.

Happy to test a patch on this setup and report the before/after counters.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.