Released snapshot aliases hand KV pages back with stale row bytes; masked attention turns them into NaN (device error 6 under KV pressure)

Đang mở
  • Có gì không ổn?
#116 5 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@mark-yong đang làm issue này rồi.

Từ ngày 3/11/2026.

  • #181 của @mark-yong — đang mở

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
48/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
backend, databases

Hướng nghiên cứu

Start with runtime_package/cache_memory.rs and scheduler/cache_memory.rs to trace how release_snapshot returns aliased pages to the KV free list, then follow the executor path that launches model work. The report's tested fix queues released pages and zeroes them on the model stream before reuse; compare that lifecycle with the single-GPU reference commit. Done means the cached-prompt reproduction no longer fails under KV pressure, including the multi-process tensor-parallel executor.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

area:caching area:kernels bug model:deepseek-v4.1 model:qwen3.8-27b priority:P1 triage:partial

Summary

When the KV pool fills, requests that resume from the prefix cache start failing with prepared model reported device error 6. The cause is stale snapshot bytes in reused KV pages:

  1. Snapshot rows borrow KV page bytes (runtime_package/cache_memory.rs).
  2. When the last owner releases a snapshot, release_snapshot returns its pages to the free list. Nothing clears the row bytes still in them.
  3. A request then receives one of those pages as its partly filled tail page. Positions past the request's last token keep the old row bytes, and in the KV format some of them decode as NaN.
  4. Attention masks those positions, but a masked position still contributes 0 × NaN = NaN to P·V. The target logits come out non-finite.
  5. PreparedStochasticVerification marks the row invalid, and the commit reports INVALID_SPECULATION (6).

It only shows up once pages cycle between snapshot rows and KV use, which is why it needs a full pool. That makes it hit long multi-turn and agent workloads, the ones the prefix cache is for.

This may also explain #61. That issue fails the same way (verification rejects the row, then the commit reports INVALID_SPECULATION), and its failing target column sits in a newly granted page. We haven't tested Gemma 4.

Environment

  • DGX Spark (GB10, SM121), single GPU, CUDA 13.0
  • nvidia/Qwen3.8-27B-NVFP4, exported with the qwen38_dense recipe defaults: 32 requests, MTP up to 5 drafts, FP8 KV
  • Flare ca1f960e plus the scratch-copy fix from 317c1ec. The code involved is unchanged on main (6256cf08).
  • flare run PACKAGE --gpus 0 with default flags (adaptive MTP verification on)

Reproduction

The repro runs on a fresh server. It sends alternating distinct ~16k- and ~32k-token prompts. Each prompt is sent once with max_tokens=1 to cache it, then again with max_tokens=64. Requests run one at a time. We used the low-entropy prompts from SPEED-Bench's throughput split.

Server First failure
--memory-limit 50G (3,760 KV pages, 101 snapshot slots) prompt 15–17, every run, about 5 minutes in
Default memory prompt 67, nine prompts after the pool first filled

The failing request is always the second (cached) send. Under --memory-limit 50G, the 101 snapshot slots' alias ranges overlap and together cover every KV page.

Repro script 
"""Feed a fresh Flare server distinct long prompts (prime with 1 token, then resend with 64) until the first failure.
Usage: repro.py OUT.csv DATA_DIR   (DATA_DIR holds SPEED-Bench throughput_16k.jsonl / throughput_32k.jsonl)"""
import csv, json, re, sys, time, urllib.error, urllib.request
out, data = sys.argv[1], sys.argv[2]
prompts = [[(ctx, r["messages"]) for r in map(json.loads, open(f"{data}/throughput_{ctx}.jsonl"))
            if r["category"] == "low_entropy"] for ctx in ("16k", "32k")]
order = [p for pair in zip(*prompts) for p in pair]
KEYS = ("flare_free_kv_tokens", "flare_cached_kv_tokens", "flare_free_snapshots", "flare_requests_failed_total")
def metrics():
    text = urllib.request.urlopen("http://127.0.0.1:8000/metrics", timeout=10).read().decode()
    return {k: float(m.group(1)) for k in KEYS for m in [re.search(rf"^{k} (\S+)$", text, re.M)] if m}
def ask(messages, max_tokens):
    body = {"model": "Qwen3.8-27B", "messages": messages, "max_tokens": max_tokens, "temperature": 1.0}
    req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    try:
        urllib.request.urlopen(req, timeout=900).read()
        return "ok"
    except urllib.error.HTTPError as e:
        return "fail: " + e.read().decode()[:120]
with open(out, "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["i", "context", "prime", "warm", "seconds", *KEYS])
    for i, (ctx, messages) in enumerate(order):
        t = time.time()
        prime, warm = ask(messages, 1), ask(messages, 64)
        m = metrics()
        w.writerow([i, ctx, prime, warm, round(time.time() - t, 1), *[m.get(k) for k in KEYS]]); f.flush()
        print(i, ctx, prime[:40], warm[:40], m, flush=True)
        if prime != "ok" or warm != "ok":
            break

Narrowing

We collected these with temporary logging in Helios and a re-exported package. None of the logging is part of this issue.

Which check fails. We made the commit report a bitmask of its shape checks in place of a bare 6. The failing round reports proposed >= 1 and accepted == proposed - 1 as failed. Verification set proposed_count = 0, so it rejected the row (speculation.py L329, L355). The first round after the restore already fails at width 1, with no drafts. That leaves the target distribution as the cause: it is non-finite, and the drafts play no part.

What we ruled out.

  • Page ownership. Before every submit we checked that each page id has exactly one owner: free list, checkpoint, retirement, request, retired request, inbound ticket or snapshot alias. There were no conflicts.
  • Reuse while in flight. We tracked, per page, the last fence that used it as KV and as snapshot backing. No page was used in one role while work in the other role was still in flight.
  • Alias range rounding. Widening every alias range by 8 pages on each side still fails.
  • Snapshot exhaustion. Free snapshots stayed above 60 throughout.
  • The row index. See the dedicated-pages rows in the table below.

What fixes it.

--memory-limit 50G, default MTP Result
As is fails at prompt 15–17
Snapshots 0–47 on dedicated pages, never handed to KV no failures through the 25-minute cap (89 prompts)
Snapshots 53–100 on dedicated pages no failures through the 25-minute cap (87 prompts)
Released alias pages zeroed before reuse no failures in 143 of 143 prompts. Default memory also passes 143 of 143, where it failed at prompt 67 before

With zeroing on, warm-request latency and benchmark throughput are unchanged. At 8k, N=1 to 32: 27.6 / 50.4 / 93.0 / 159 / 242 / 305 tok/s, against 28.2 / 49.0 / 92.8 / 154 / 244 / 308 tok/s without it.

Why only the tail page matters. _copy_pages (state_transfer.py L153) saves and restores the request's whole tail page, including positions past the endpoint. Startup zeroes every cache byte once (runtime_package/cache_memory.rs L478, "before any graph or transfer can read an alias"). After that, nothing clears a page when it goes from row storage back to the KV pool (scheduler/cache_memory.rs L153–165).

Potential fix

These are three options. The first is the one we tested.

  1. Zero pages leaving alias use. When release_snapshot drops a page's alias count to zero, queue the page. The executor then zeroes it on the model stream before the next graph launch, so the clear happens after the snapshot's last reader and before any KV writer. A single-GPU version of this is on a fork branch: shadowpa0327/flare-local-inference@b83d347a. It uses a global page list and host-to-device copies from pinned zeros, and doesn't cover the multi-process tensor-parallel executor. Treat it as a reference, not a proposed patch.
  2. Clear only the dead tail. When a page is granted to a request, or a tail page is restored, zero the positions past the endpoint. This moves less data than option 1, but every writer of a partial page has to honor it.
  3. Make masked positions inert in attention. Skip or zero masked K/V positions in the paged kernels, instead of relying on 0 × value staying finite. This also protects any other path that leaves undefined bytes in a page.

Separately, as #61 also notes, a row the verifier rejects reaches the commit as INVALID_SPECULATION. Reporting it as INVALID_DISTRIBUTION would have pointed at the cause much sooner. The state-transfer validator also uses 6, for an unmapped page, which sent us looking at page tables first.

Found and written up with Claude Code (Claude Opus 5.5) while benchmarking Flare on DGX Spark.

Ngôn ngữ chính
Rust
Star
800
Fork
106
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
100

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/hyperflare-inference

Tất cả issue của NVIDIA/hyperflare-inference

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.