Proposal: dense Qwen3.8-27B DFlash2 with BF16 and calibrated NVFP4 drafters

Read the linked experimental branch and compare its DFlash2 addition with the existing parallel-draft speculative transaction and the native MTP proposal in #20. Start by checking the stated integration gates: current-main integration, the open FlashInfer fix in #14/#43, and the missing Nemotron GPU regression and reference parity controls. Done means maintainers agree on the integration path and the required gates are resolved before a recipe PR; the issue identifies no project file paths or test commands.

Proposal and requested maintainer decision

Would you like a focused dense-Qwen DFlash2 contribution, following the native MTP proposal in #20? There is a working experimental implementation with BF16 and independently calibrated NVFP4 drafters on one RTX PRO 4500 Blackwell (SM120, 32GB). I am offering the code for discussion rather than presenting the older experimental branch as a current-main integration.

Code: https://github.com/Maxrio9/flare-local-inference/tree/qwen38-dense-dflash2-20261023 (two feature commits squashed over the already-public Maxrio9:qwen38-dense-native-mtp stack). The DFlash2 addition preserves the existing dense alias. New aliases have conservative C1/32K/prefill512 defaults and fixed seven drafts. Private C4/64K exports received broader qualification; a 128K export passed capacity checks only.

Pinned checkpoints:

• target nvidia/Qwen3.8-27B-NVFP4@482ca0f3832238542f8f5295dde86b5f22711d80;
• BF16 drafter z-lab/Qwen3.8-27B-DFlash2@50307d4c4cde6860d4eee73e2547cd786fe8e8a4;
• calibrated drafter maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal@bd7a934213c47a9e7ef69eef36bb3325f47fd1f1.

Integration and semantics

Reuses the existing parallel-draft speculative transaction, GDN verification/commit, shared grouped convolution, stochastic verifier and state-transfer contract. Target features are pre-final-norm outputs of zero-based layers 5/19/33/47/61. Context has lag zero and a complete five-layer 2048-token ring, including snapshots; only committed, EOS/budget-clipped rows enter it. Target KV is FP8; draft KV/ring is BF16, including with the NVFP4 drafter. No FP8 draft-cache optimization is claimed.

The 35 calibrated NVFP4 projections retain independent checkpoint scales, including per-layer context K/V scales. The other 46 BF16 tensors match the pinned BF16 drafter byte-for-byte; shared scalar groups were checked equal. No runtime requantization. Package identity includes the actual drafter payload SHA-256.

Proposal q is the realized predecessor-conditioned top-16 selector distribution after Flare's sampling filters/penalties, retained for stochastic verification. The reference uses temperature-only q and BF16 selector scores, while this implementation computes selector scores in FP32; published acceptance figures are therefore not parity claims. Missing-predecessor rows become invalid, rather than silently choosing candidate zero. Embeddings remain the target's GPU-resident BF16 table; the RAM-table flag does not offload this dense table.

The branch lifts shared DFlash execution pieces from Nemotron for reuse. It proposes no new captured-operation family, Protobuf or model-specific native policy; if review finds a changed operation contract is required, that needs the project owner's explicit approval first.

Results and limits

Official target, native C1 per-token decode rates, medians of three seeds per authored case, natural EOS, 2048-token budgets. MTP3 and MTP7 are fixed-width arms with adaptive verification disabled:

Chat/code/agent: thinking off, T0.7/top-p0.8/top-k20/presence1.5; reasoning: thinking on, T1/top-p0.95/top-k20/presence0. No reasoning-effort value requested. 60/60 transport/runtime completions. Different stochastic answers and lengths mean these are not equal-work latency factors or quality-equivalence evidence. They are results for earlier frozen experimental packages, not benchmarks of current main or the final memory branch.

Nsight B1/W8 kernel-sum phases in ms (pre-target / target / post-target): MTP 14.17/26.74/0.98; BF16 DFlash 5.62/26.66/1.29; NVFP4 DFlash 3.10/26.66/1.22. Corresponding node spans were 43.16/34.71/31.92ms. Sums may overlap; no summation as elapsed time, tracing-overhead or replay-floor claim. The two DFlash vocabulary projections share the quantized target head. These are short-context traced replays, not native-counter timing of the completed-answer run.

A controlled companion screen on the all-NVFP4 target used 18 frozen prefixes (one plain trajectory per category, 16 seeds/prefix). DFlash2 did not beat MTP7 on chat acceptance: BF16/NVFP4 DFlash emitted 3.20/3.325 tokens per round versus MTP7 3.462. Agent emitted 2.475/2.513 versus 2.350. These are conditional acceptance observations, not independent quality trials or official-target measurements. Cheaper W8 rounds are an important part of the chat/agent throughput result; higher throughput does not establish higher acceptance, and the completed-answer text differences prevent assigning the entire gap to round cost.

The NVFP4 drafter saves 2,402,382,848 retained weight bytes (2.237 GiB) versus BF16; each full draft window costs 40 MiB per saved state. On the exact offered feature tree, 65 CPU structural/CLI/checkpoint-lineage tests passed with both pinned checkpoint paths supplied. Public CLI export exercised all seven depths (175 requests/reuse probes), with state/restore, multi-chunk scored-prefill, padded/inactive C4 and native pressure tests on the experimental stack. 51K full-prefix reuse passed; 65,300+128 and 130,800+128-token fixtures passed in plain/fixed7. The 128K test uses repetitive prompts and forced output budgets: capacity, not long-context quality or full pressure qualification.

Known gates before a recipe PR:

• #20's prior MTP/loading/accounting stack needs maintainer agreement and current-main integration; #14 / local-inference-lab/flashinfer#43 is still open. All measured packages used FlashInfer03de0b66 plus that PV-tail fix, not the current unpatched lock.
• Full Nemotron-checkpoint GPU regression for the extracted shared modules is unrun. Independent HF tap +/-1 and complete reference window/convolution negative controls are not established by structural/state tests.
• An optional longer 22-prefix parity run stopped because its reference fixture backed only1280 tokens for a1300-token prefix. The corrected1536-token fixture was not rerun. Completed native128K capacity is a separate test.
• No semantic-quality equivalence claim: the96-request official authored protocol suite completed, with23 semantic fixture failures across plain/DFlash (budgets/literal checks included).
• DSpark is research only. Additional memory changes and scheduler fixes are separate submissions, not bundled here.

Code authored by Codex (GPT-6), AI-assisted. Claude reviewed recipe semantics, tests and reporting within AGENTS scope; no Claude Opus kernel/fusion/packing/overlap/scratch work is incorporated. Hardware evidence is SM120 only; no SM121/TP qualification.