Host prefix cache: ~30 % of hits dropped as capacity_rejected and fully recomputed

Abierto
  • ¿Algo no está bien?
#255 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
42/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend, performance

Línea de trabajo

Start with the host-cache hit admission path in helios/src/server/runtime.rs and resource checks in helios/src/scheduler.rs. Trace how ensure_resources distinguishes pages awaiting GPU retirement from exhausted live resources, then inspect the related admission and host-cache tests. Done means capacity rejection causes are observable or a retry/partial-restore behavior is implemented and covered by tests.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Setup
  • Flare 972e28e, built from source with one metrics-only patch (adds a model_name label to /metrics).
  • DeepSeek V4.1 Flash 1017 (gb202 payload package), 4 × RTX PRO 6000 Blackwell (SM120), TP4.
  • --scheduler-policy throughput --mtp-draft-tokens 7 --mtp-adaptive-verification --prefix-cache-memory 60G --prefix-cache-disk <dir> --prefix-cache-disk-limit 800G
  • Traffic: agentic coding sessions (long shared prefixes, prompts up to ~165k tokens), ~21k requests/day.
What happens

When a request's prefix is found in the host cache, roughly 30 % of those hits are thrown away because the restore does not fit in GPU KV pages at that moment. The request is then admitted without the restore and the whole prefix is prefilled again.

Last 24 h, flare_host_cache_lookup_outcomes_total:

reason count
no_match 14023
admitted 4068
capacity_rejected 1724
cost_rejected 73

So 1724 of 5792 usable hits (30 %) end up as full recomputation, often of long prefixes. That is exactly the work the host cache exists to avoid.

Where it comes from

helios/src/server/runtime.rs, admission with a host-cache hit:

let admitted = match scheduler.admit_planned(spec.clone(), plan, Some(inbound)) {
    Ok(admitted) => admitted,
    Err(Error::Admission(AdmissionError::Capacity(_))) => {
        let handle = scheduler.admit(spec)?;
        scheduler.record_host_lookup();
        return Ok((handle, Some(LookupOutcome::CapacityRejected)));
    }
    ...

ensure_resources (helios/src/scheduler.rs) returns Capacity for two different reasons:

  • "request resources are awaiting GPU retirement": enough pages exist but are still fenced.
  • "live request resources are exhausted or awaiting GPU retirement": nothing more can be evicted.

In both cases the hit is dropped right away. The plain admit(spec) fallback then succeeds, so the request was admissible. It just could not take all the restore pages at once.

The rejections don't only happen when the pool is empty. Bucketing the 24 h by the minimum of flare_free_kv_pages in each 5-minute window:

min free KV pages (5 min) capacity_rejected
0–10 495
11–100 332
101–1000 417
> 1000 480

(The gauge is sampled, so short dips may be missed. Still, a good share happens with plenty of free pages, which points at fenced pages awaiting retirement and/or restores larger than the instantaneous free set.)

Suggestions (any of these would help)
  1. Split the outcome: count the two Capacity reasons separately (capacity_pending_retirement vs capacity_exhausted), so operators can tell them apart.
  2. Defer instead of dropping on "awaiting GPU retirement": keep the request queued with its hit and retry admission once the fenced pages retire, rather than falling back to a full prefill.
  3. Partial restore: if only part of the hit fits, restore the longest prefix that fits and prefill the rest.

Happy to test a patch on this setup and report the before/after counters.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.