Qwen3.8 `--mixed-prefill` exports fail on main: two regressions (317c1ecc, bce032c0)

Aperta
  • Qualcosa non va?
#115 2 commenti 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
backend, infrastructure

Direzione di ricerca

Start with ignition/recipes/qwen38/mixed_model.py and ignition/recipes/qwen38/decoder.py to trace why mixed verification supplies fewer sample rows than hidden rows; compare that with ignition/preparation/execution.py and heliopause/src/export/binding.rs for proposal depth. Reproduce with the Qwen3.8 --mixed-prefill export command in the issue, then check that graph preparation and native export both complete and the package loads.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:preparation bug model:qwen3.8-flash-next priority:P1 triage:review
Summary

Exporting Qwen3.8 Flash-Next with --mixed-prefill fails on current main (6256cf08). There are two separate regressions:

  1. Since 317c1ecc (Oct 26): preparation stops at the first mixed graph with ValueError: verification requires one sample row per hidden row.
  2. Since bce032c0 (Oct 24): preparation completes, but the native export fails at its last step with speculative graph lacks explicit proposal depth. This is still the case at ca1f960e, the parent of 317c1ecc. It shows up only after the full preparation, about 75 minutes on a GB10.

The same command exported and served fine at 1fd1f917 (Oct 23). I have not tried exports without --mixed-prefill on these commits.

Setup
  • 1× DGX Spark (GB10, SM121), driver 580.159.03, CUDA 13.0, kernel 6.17.0-1026-nvidia
  • Checkpoint nvidia/Qwen3.8-Flash-Next-FP4-Local @ b9ce18a6
  • One unrelated local change: the Qwen recipe's top_k sampling range is widened to 1–20 in ignition/recipes/qwen38/recipe.py.
ignition export qwen3.8-flash --gpus 0 --max-concurrency 8 --image-concurrency 4 \
  --mtp-draft-tokens 5 --speculative-token-capacity 48 --mixed-prefill --verbose
1. verification requires one sample row per hidden row (since 317c1ecc)

At 7c1a9fde, after about 2 minutes:

[  FAILED  ] Construct graph catalog.  52/159  91.8s
  File "ignition/recipes/qwen38/bootstrap.py", line 447, in _prepare_model
    operation = PreparedMixedRound(mixed_owner, batch, width, prefill, name,
  File "ignition/recipes/qwen38/mixed_model.py", line 87, in __init__
    self.round = PreparedSpeculativeRound(decode_owner, batch, width, name,
  File "ignition/recipes/qwen38/speculative_model.py", line 166, in __init__
    self.decoder = decoder_factory(owner.weights, scratch, self.state, owner.cache, owner.rotary,
  File "ignition/recipes/qwen38/mixed_model.py", line 85, in speculative_decoder
    view = decoder_factory(*args, **kwargs)
  File "ignition/recipes/qwen38/mixed_model.py", line 69, in decoder_factory
    self.decoder = PreparedDecoder(weights, full_scratch, state, cache, rotary, table, workspaces,
  File "ignition/recipes/qwen38/decoder.py", line 223, in __init__
    raise ValueError("verification requires one sample row per hidden row")
ValueError: verification requires one sample row per hidden row

317c1ecc added this check to PreparedDecoder and aliases sample_hidden to hidden when verification=True (decoder.py#L221-L224). The mixed round's decoder packs the prompt and decode tokens into self.packing.tokens hidden rows. It samples only batch * width + 1 of them, the decode rows plus the prompt's last row, and passes verification=True (mixed_model.py#L65-L70). So the shapes never match for a mixed graph. At ca1f960e the graph catalog completes (159/159).

2. speculative graph lacks explicit proposal depth (since bce032c0)

At ca1f960e, every preparation phase passes: graph catalog, kernel tuning, compile, warm-up and graph capture. Then:

[  FAILED  ] Export native package.  8.9s
  File "ignition/preparation/runtime_package.py", line 134, in _capture
    self.native_output = flare_heliopause.export_capture(json.dumps(request), owners, recorder)
RuntimeError: speculative graph lacks explicit proposal depth
error: native package export failed: speculative graph lacks explicit proposal depth

The two sides disagree about mixed graphs:

  • Heliopause requires draft_tokens when kind == "speculative" || kind == "mixed" && speculative (binding.rs#L188-L194).
  • Ignition never provides one for a mixed graph:

Before bce032c0, binding.rs filled in draft_tokens = width - 1 for these graphs, and the loader still expects that. For a mixed graph with speculation enabled, selection.rs:

Dropping the mixed case from the exporter check is not enough. The export then succeeds, but flare run fails at "Prepare model" with invalid or duplicate serving graph variant (selection.rs#L236).

Workaround I tested

Restore the earlier default in the exporter, for mixed graphs only:

--- a/heliopause/src/export/binding.rs
+++ b/heliopause/src/export/binding.rs
@@ -187,7 +187,11 @@ pub(crate) fn execution(
             .unwrap_or(false);
         if kind == "speculative" || kind == "mixed" && speculative {
             if !graph.contains_key("draft_tokens") {
-                return Err(super::invalid("speculative graph lacks explicit proposal depth"));
+                // Mixed graphs carry no explicit depth from preparation; use the pre-ABI4 default.
+                if kind != "mixed" {
+                    return Err(super::invalid("speculative graph lacks explicit proposal depth"));
+                }
+                graph.insert("draft_tokens".into(), width.saturating_sub(1).into());
             }
             graph.insert("verify_tokens".into(), width.into());
             graph.insert("draft_source".into(), "MTP".into());

Results at ca1f960e with this change:

  • Export and load: the export completes and the package loads: 76.5 GiB GPU, 1.79M KV tokens.
  • Serving: flare run … --no-mtp-adaptive-verification --mtp-draft-tokens 3 served without errors through:
    • two runs of 8 concurrent multi-turn sessions (48 turns each), with flare_graph_launches_total{kind="mixed"} at 24 and 53;
    • single-request decode and 32K-token prompt probes.

A cleaner fix may belong on the Ignition side: let a mixed graph with speculation declare its depth (width - 1 in the Qwen recipe), so the exporter check can stay strict.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.