Proposal: isolate logprob reads (max_tokens 1) from batch composition and the prefix cache so scoring is reproducible

オープン
  • 何か問題がありますか？
#772 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
40/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python, pytorch
領域
backend, performance

調査の方向性

Start by reading scheduler/prefill.py around PrefillManager.schedule_next_batch and PrefillAdder, then trace prefix-cache matching and donation. Implement the opt-in --logprob-read-isolate behavior for reads with logprobs set and max_tokens == 1, including chunked continuations. Done means isolated reads bypass and do not populate the prefix cache, while chat and multi-token requests retain their current path; reproduce the bitwise-equality cases described in the issue.

索引モデルが issue の本文から書いたものです。

説明

What happened

A scoring request (one forward, max_tokens: 1, read the next-token logprobs) gets different numbers depending on which other requests happen to share its prefill batch, and on what the prefix cache holds. Classifier-style clients (AnyJev's VLLMBackend and Tacit's ServerEngine; see #768) ask the same question in several option orders and compare the scores. So the answer moves with unrelated traffic.

Measured on Qwen3.8-Flash-Next-NVFP4 with logprobs (#224) applied. The requests use temperature: 0, max_tokens: 1, logprobs: 5, and prompts of about 60 tokens:

top-5 logprob deviation vs the same prompt sent alone
3 prompts in one /v1/completions request (2 prefilled together) 0.5-0.87 nats
8 different prompts in parallel 7 of 8 differ
the same prompt sent again (prefix-cache hit: 256 of 300 tokens cached) differs
the same prompt after a chat turn on the same prefix differs

The top-1 token did not flip in these cases, but the gaps between options that the scores are compared on did change.

Cause (current main, cc56b6bd)
  • Batch shape. The first divergence between a row prefilled alone and the same row prefilled with another sequence is the bf16 hyper-connection down GEMM (models/qwen4_exp/hc.py:119, K=10240, N=336). cuBLAS picks a different kernel for a different total M, the result differs by 1 ulp, and over 48 layers that grows to the deviations above. With M held fixed and only the partner's content changed, the rows stay bitwise equal, so this is kernel selection, not contamination.
  • Prefix cache. A hit resumes from KV and a GDN snapshot computed under a different batch shape, which #748 describes.
  • Scheduling. PrefillManager.schedule_next_batch (scheduler/prefill.py:294-315) packs any pending requests together, and PrefillAdder (:76) always matches the prefix cache.
Proposal

An opt-in server flag, --logprob-read-isolate. A "read" is a request with logprobs set and max_tokens == 1. Under the flag:

  1. A read is admitted only as the first request of a prefill pass and closes that pass. Its chunked continuation is also alone.
  2. A read matches the prefix cache with max_len=0 and does not donate to it at the end.

Chat and multi-token requests keep the current path, so decode is unchanged.

With this in place on the setup above:

  • Exactness: every case in the table becomes bitwise equal, for repeats at 300, 512, 513, 819, 820 and 4096 tokens, after a chat turn, and with 8 reads in parallel. Greedy chat output is unchanged.
  • Cost: short parallel reads run about 1.8x slower (8 reads of 60 tokens: 1.48 s, now 2.62 s). Scoring that rotates options over a shared prefix runs about 2x slower, because it no longer reuses the prefix.
  • Side effect: scoring bursts no longer evict a long conversation's GDN snapshot. After 96 reads, a 38K-token conversation resumed in 0.54 s, against 13.9 s for a full recompute without the flag.

This does not make chat batch-invariant. That would need batch-invariant GEMM configurations (#748 asks for a deterministic mode).

Environment

RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130, nvidia/Qwen3.8-Flash-Next-NVFP4, main cc56b6bd with #224 applied.

主要言語
Python
スター
14.2k
フォーク
1.4k
平均マージ
18時間 46分
マージ済み PR（30日）
98

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

FlashML-org/FreeToken のほかの issue

FlashML-org/FreeToken の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。