Qwen3.8 Flash Next DFlash2: mixed prefill rounds, YaRN serving, and served measurements from a fork

Aperta
  • Qualcosa non va?
#210 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Raydius ci sta già lavorando.

Dal 5/11/2026.

  • #211 di @Raydius — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
20/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend, devtools

Direzione di ricerca

Start with qwen38/bootstrap.py, components/dflash2.py, and the mixed-round graph signature and binding paths referenced in the issue. Reproduce the reported export failure with the CPU tests, then inspect the fork's mixed speculative package changes and served measurements. Done means the export defect is fixed and maintainers have decided whether mixed DFlash2 rounds and YaRN serving fit the existing contracts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

model:qwen3.8-flash-next triage:design-review

d68ed8f5 added shared DFlash2 preparation for Qwen3.8 Flash Next on components/dflash2.py, selected with --speculative-method dflash2. A fork (Raydius/flare-local-inference, branch wt/dflash-s-e2vk) has served Flash Next with the incoai/Qwen3.8-Flash-Next-DFlash2 draft on an RTX PRO 6000 (SM120, TP1) since 2026-10-31 and, after d68ed8f5, the fork's own recipe is redundant except for three things. Before opening PRs for any of them I would like your decision on each, since AGENTS.md asks for an explicit maintainer request and for prior approval of any new captured-graph operation contract.

  1. Mixed prefill rounds with a DFlash2 draw. Upstream's qwen38/bootstrap.py rejects the combination ("DFlash2 requires separate prefill and verification graphs"). The fork prepares mixed prefill+decode rounds whose embedded decode round draws from DFlash2: the same mixed graph family (one prompt row plus b decode rows at width w, selected by the manifest's mixed depth): the decode half composes the shared proposal-only DFlash2 round around the packed target, and the prompt half, built the way the MTP absorption is, writes the chunk's captured features into its request's context rings after the round. Exported and served under --mixed-prefill --max-concurrency 8 with prompt-cache reuse; no new graph kind, input binding, or Helios policy. Question: is a mixed round that embeds a DFlash2 draw a shape variant of the existing mixed family (an operation-contract-preserving change, PR-able as a recipe change), or a new operation contract that needs your approval first?
  2. YaRN serving for the 1M-token Flash Next context. The fork carries two small patches: a qwen3.8-flash-yarn4 checkpoint declaration (context=1_048_576, RopeScaling(4., 262144), the same weights) and --rope-parameters (frequency-only interpolation without YaRN's attention temperature, applied to any resolved checkpoint and recorded in the plan). With DFlash2 the draft's rotary and its 2048-position rings take the extended positions unscaled; a probe past 262k tokens kept 3.69 accepted tokens per round (acceptance rate 0.618) against 2.11 / 0.699 below it. Question: do you want that as a PR (declaration first, then the flag), or should long context stay out of the Flash Next recipe?
  3. Served measurements, in case they help set the DFlash2 defaults. All from flare run of exported packages (never debug-serve), 28-case frozen code-edit workload, single trials with about ±8 % repeatability, history copies off:
arm (serve-time width on an adaptive package) c1 tok/s c1 accepted/round c8 tok/s c8 accepted/round
adaptive 405.5 1.85 533.2 3.94
fixed-0 (no speculation) 153.8 – 424.5 –
fixed-3 360.6 1.74 459.6 3.42
fixed-7 537.3 2.44 351.7 3.87

A package exported fixed-7 (--no-mtp-adaptive-verification, only fixed-width rounds captured) served 565.5 tok/s at c1 and 417.0 at c8 (acceptance rate 0.587 / 0.606). Single-stream throughput rises with every extra draft up to 7; at eight streams the fixed curve peaks at 3 and the adaptive controller beats every fixed width.

Independently of the above, a defect PR is coming first: since 317c1ecc and bce032c0 no package with mixed rounds that also speculate can be exported (the mixed decoder is built in verification mode, which now aliases sample rows to hidden rows, and GraphSignature forbids draft_tokens on mixed kinds while the binding requires it); upstream's own Qwen3.8 MTP mixed export is affected. Two commits, CPU tests failing on the parent, a mixed speculative package exported and served on SM120.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.