Host prefix cache: ~30 % of hits dropped as capacity_rejected and fully recomputed

オープン
  • 何か問題がありますか？
#255 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
42/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
rust
領域
backend, performance

調査の方向性

Start with the host-cache hit admission path in helios/src/server/runtime.rs and resource checks in helios/src/scheduler.rs. Trace how ensure_resources distinguishes pages awaiting GPU retirement from exhausted live resources, then inspect the related admission and host-cache tests. Done means capacity rejection causes are observable or a retry/partial-restore behavior is implemented and covered by tests.

索引モデルが issue の本文から書いたものです。

説明

Setup
  • Flare 972e28e, built from source with one metrics-only patch (adds a model_name label to /metrics).
  • DeepSeek V4.1 Flash 1017 (gb202 payload package), 4 × RTX PRO 6000 Blackwell (SM120), TP4.
  • --scheduler-policy throughput --mtp-draft-tokens 7 --mtp-adaptive-verification --prefix-cache-memory 60G --prefix-cache-disk <dir> --prefix-cache-disk-limit 800G
  • Traffic: agentic coding sessions (long shared prefixes, prompts up to ~165k tokens), ~21k requests/day.
What happens

When a request's prefix is found in the host cache, roughly 30 % of those hits are thrown away because the restore does not fit in GPU KV pages at that moment. The request is then admitted without the restore and the whole prefix is prefilled again.

Last 24 h, flare_host_cache_lookup_outcomes_total:

reason count
no_match 14023
admitted 4068
capacity_rejected 1724
cost_rejected 73

So 1724 of 5792 usable hits (30 %) end up as full recomputation, often of long prefixes. That is exactly the work the host cache exists to avoid.

Where it comes from

helios/src/server/runtime.rs, admission with a host-cache hit:

let admitted = match scheduler.admit_planned(spec.clone(), plan, Some(inbound)) {
    Ok(admitted) => admitted,
    Err(Error::Admission(AdmissionError::Capacity(_))) => {
        let handle = scheduler.admit(spec)?;
        scheduler.record_host_lookup();
        return Ok((handle, Some(LookupOutcome::CapacityRejected)));
    }
    ...

ensure_resources (helios/src/scheduler.rs) returns Capacity for two different reasons:

  • "request resources are awaiting GPU retirement": enough pages exist but are still fenced.
  • "live request resources are exhausted or awaiting GPU retirement": nothing more can be evicted.

In both cases the hit is dropped right away. The plain admit(spec) fallback then succeeds, so the request was admissible. It just could not take all the restore pages at once.

The rejections don't only happen when the pool is empty. Bucketing the 24 h by the minimum of flare_free_kv_pages in each 5-minute window:

min free KV pages (5 min) capacity_rejected
0–10 495
11–100 332
101–1000 417
> 1000 480

(The gauge is sampled, so short dips may be missed. Still, a good share happens with plenty of free pages, which points at fenced pages awaiting retirement and/or restores larger than the instantaneous free set.)

Suggestions (any of these would help)
  1. Split the outcome: count the two Capacity reasons separately (capacity_pending_retirement vs capacity_exhausted), so operators can tell them apart.
  2. Defer instead of dropping on "awaiting GPU retirement": keep the request queued with its hit and retry admission once the fenced pages retire, rather than falling back to a full prefill.
  3. Partial restore: if only part of the hit fits, restore the longest prefix that fits and prefill the rest.

Happy to test a patch on this setup and report the before/after counters.

主要言語
Rust
スター
800
フォーク
106
平均マージ
1日 3時間
マージ済み PR（30日）
100

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/hyperflare-inference のほかの issue

NVIDIA/hyperflare-inference の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。