Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure)

Aperta
  • Qualcosa non va?
#116 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@mark-yong ci sta già lavorando.

Dal 3/11/2026.

  • #181 di @mark-yong — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
backend, databases

Direzione di ricerca

Start with runtime_package/cache_memory.rs and scheduler/cache_memory.rs to trace how release_snapshot returns aliased pages to the KV free list, then follow the executor path that launches model work. The report's tested fix queues released pages and zeroes them on the model stream before reuse; compare that lifecycle with the single-GPU reference commit. Done means the cached-prompt reproduction no longer fails under KV pressure, including the multi-process tensor-parallel executor.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:caching area:kernels bug model:deepseek-v4.1 model:qwen3.8-27b priority:P1 triage:partial

Summary

When the KV pool fills, requests that resume from the prefix cache start failing with prepared model reported device error 6. The cause is stale snapshot bytes in reused KV pages:

  1. Snapshot rows borrow KV page bytes (runtime_package/cache_memory.rs).
  2. When the last owner releases a snapshot, release_snapshot returns its pages to the free list. Nothing clears the row bytes still in them.
  3. A request then receives one of those pages as its partly filled tail page. Positions past the request's last token keep the old row bytes, and in the KV format some of them decode as NaN.
  4. Attention masks those positions, but a masked position still contributes 0 × NaN = NaN to P·V. The target logits come out non-finite.
  5. PreparedStochasticVerification marks the row invalid, and the commit reports INVALID_SPECULATION (6).

It only shows up once pages cycle between snapshot rows and KV use, which is why it needs a full pool. That makes it hit long multi-turn and agent workloads, the ones the prefix cache is for.

This may also explain #61. That issue fails the same way (verification rejects the row, then the commit reports INVALID_SPECULATION), and its failing target column sits in a newly granted page. We haven't tested Gemma 4.

Environment

  • DGX Spark (GB10, SM121), single GPU, CUDA 13.0
  • nvidia/Qwen3.8-27B-NVFP4, exported with the qwen38_dense recipe defaults: 32 requests, MTP up to 5 drafts, FP8 KV
  • Flare ca1f960e plus the scratch-copy fix from 317c1ec. The code involved is unchanged on main (6256cf08).
  • flare run PACKAGE --gpus 0 with default flags (adaptive MTP verification on)

Reproduction

The repro runs on a fresh server. It sends alternating distinct ~16k- and ~32k-token prompts. Each prompt is sent once with max_tokens=1 to cache it, then again with max_tokens=64. Requests run one at a time. We used the low-entropy prompts from SPEED-Bench's throughput split.

Server First failure
--memory-limit 50G (3,760 KV pages, 101 snapshot slots) prompt 15–17, every run, about 5 minutes in
Default memory prompt 67, nine prompts after the pool first filled

The failing request is always the second (cached) send. Under --memory-limit 50G, the 101 snapshot slots' alias ranges overlap and together cover every KV page.

Repro script 
"""Feed a fresh Flare server distinct long prompts (prime with 1 token, then resend with 64) until the first failure.
Usage: repro.py OUT.csv DATA_DIR   (DATA_DIR holds SPEED-Bench throughput_16k.jsonl / throughput_32k.jsonl)"""
import csv, json, re, sys, time, urllib.error, urllib.request
out, data = sys.argv[1], sys.argv[2]
prompts = [[(ctx, r["messages"]) for r in map(json.loads, open(f"{data}/throughput_{ctx}.jsonl"))
            if r["category"] == "low_entropy"] for ctx in ("16k", "32k")]
order = [p for pair in zip(*prompts) for p in pair]
KEYS = ("flare_free_kv_tokens", "flare_cached_kv_tokens", "flare_free_snapshots", "flare_requests_failed_total")
def metrics():
    text = urllib.request.urlopen("http://127.0.0.1:8000/metrics", timeout=10).read().decode()
    return {k: float(m.group(1)) for k in KEYS for m in [re.search(rf"^{k} (\S+)$", text, re.M)] if m}
def ask(messages, max_tokens):
    body = {"model": "Qwen3.8-27B", "messages": messages, "max_tokens": max_tokens, "temperature": 1.0}
    req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    try:
        urllib.request.urlopen(req, timeout=900).read()
        return "ok"
    except urllib.error.HTTPError as e:
        return "fail: " + e.read().decode()[:120]
with open(out, "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["i", "context", "prime", "warm", "seconds", *KEYS])
    for i, (ctx, messages) in enumerate(order):
        t = time.time()
        prime, warm = ask(messages, 1), ask(messages, 64)
        m = metrics()
        w.writerow([i, ctx, prime, warm, round(time.time() - t, 1), *[m.get(k) for k in KEYS]]); f.flush()
        print(i, ctx, prime[:40], warm[:40], m, flush=True)
        if prime != "ok" or warm != "ok":
            break

Narrowing

We collected these with temporary logging in Helios and a re-exported package. None of the logging is part of this issue.

Which check fails. We made the commit report a bitmask of its shape checks in place of a bare 6. The failing round reports proposed >= 1 and accepted == proposed - 1 as failed. Verification set proposed_count = 0, so it rejected the row (speculation.py L329, L355). The first round after the restore already fails at width 1, with no drafts. That leaves the target distribution as the cause: it is non-finite, and the drafts play no part.

What we ruled out.

  • Page ownership. Before every submit we checked that each page id has exactly one owner: free list, checkpoint, retirement, request, retired request, inbound ticket or snapshot alias. There were no conflicts.
  • Reuse while in flight. We tracked, per page, the last fence that used it as KV and as snapshot backing. No page was used in one role while work in the other role was still in flight.
  • Alias range rounding. Widening every alias range by 8 pages on each side still fails.
  • Snapshot exhaustion. Free snapshots stayed above 60 throughout.
  • The row index. See the dedicated-pages rows in the table below.

What fixes it.

--memory-limit 50G, default MTP Result
As is fails at prompt 15–17
Snapshots 0–47 on dedicated pages, never handed to KV no failures through the 25-minute cap (89 prompts)
Snapshots 53–100 on dedicated pages no failures through the 25-minute cap (87 prompts)
Released alias pages zeroed before reuse no failures in 143 of 143 prompts. Default memory also passes 143 of 143, where it failed at prompt 67 before

With zeroing on, warm-request latency and benchmark throughput are unchanged. At 8k, N=1 to 32: 27.6 / 50.4 / 93.0 / 159 / 242 / 305 tok/s, against 28.2 / 49.0 / 92.8 / 154 / 244 / 308 tok/s without it.

Why only the tail page matters. _copy_pages (state_transfer.py L153) saves and restores the request's whole tail page, including positions past the endpoint. Startup zeroes every cache byte once (runtime_package/cache_memory.rs L478, "before any graph or transfer can read an alias"). After that, nothing clears a page when it goes from row storage back to the KV pool (scheduler/cache_memory.rs L153–165).

Potential fix

These are three options. The first is the one we tested.

  1. Zero pages leaving alias use. When release_snapshot drops a page's alias count to zero, queue the page. The executor then zeroes it on the model stream before the next graph launch, so the clear happens after the snapshot's last reader and before any KV writer. A single-GPU version of this is on a fork branch: shadowpa0327/flare-local-inference@b83d347a. It uses a global page list and host-to-device copies from pinned zeros, and doesn't cover the multi-process tensor-parallel executor. Treat it as a reference, not a proposed patch.
  2. Clear only the dead tail. When a page is granted to a request, or a tail page is restored, zero the positions past the endpoint. This moves less data than option 1, but every writer of a partial page has to honor it.
  3. Make masked positions inert in attention. Skip or zero masked K/V positions in the paged kernels, instead of relying on 0 × value staying finite. This also protects any other path that leaves undefined bytes in a page.

Separately, as #61 also notes, a row the verifier rejects reaches the commit as INVALID_SPECULATION. Reporting it as INVALID_DISTRIBUTION would have pointed at the cause much sooner. The state-transfer validator also uses 6, for an unmapped page, which sent us looking at page tables first.

Found and written up with Claude Code (Claude Opus 5.5) while benchmarking Flare on DGX Spark.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.