Qwen3.8 `--mixed-prefill` exports fail on main: two regressions (317c1ecc, bce032c0)

Abierto
  • ¿Algo no está bien?
#115 2 comentarios 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, rust
Área
backend, infrastructure

Línea de trabajo

Start with ignition/recipes/qwen38/mixed_model.py and ignition/recipes/qwen38/decoder.py to trace why mixed verification supplies fewer sample rows than hidden rows; compare that with ignition/preparation/execution.py and heliopause/src/export/binding.rs for proposal depth. Reproduce with the Qwen3.8 --mixed-prefill export command in the issue, then check that graph preparation and native export both complete and the package loads.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area:preparation bug model:qwen3.8-flash-next priority:P1 triage:review
Summary

Exporting Qwen3.8 Flash-Next with --mixed-prefill fails on current main (6256cf08). There are two separate regressions:

  1. Since 317c1ecc (Oct 26): preparation stops at the first mixed graph with ValueError: verification requires one sample row per hidden row.
  2. Since bce032c0 (Oct 24): preparation completes, but the native export fails at its last step with speculative graph lacks explicit proposal depth. This is still the case at ca1f960e, the parent of 317c1ecc. It shows up only after the full preparation, about 75 minutes on a GB10.

The same command exported and served fine at 1fd1f917 (Oct 23). I have not tried exports without --mixed-prefill on these commits.

Setup
  • 1× DGX Spark (GB10, SM121), driver 580.159.03, CUDA 13.0, kernel 6.17.0-1026-nvidia
  • Checkpoint nvidia/Qwen3.8-Flash-Next-FP4-Local @ b9ce18a6
  • One unrelated local change: the Qwen recipe's top_k sampling range is widened to 1–20 in ignition/recipes/qwen38/recipe.py.
ignition export qwen3.8-flash --gpus 0 --max-concurrency 8 --image-concurrency 4 \
  --mtp-draft-tokens 5 --speculative-token-capacity 48 --mixed-prefill --verbose
1. verification requires one sample row per hidden row (since 317c1ecc)

At 7c1a9fde, after about 2 minutes:

[  FAILED  ] Construct graph catalog.  52/159  91.8s
  File "ignition/recipes/qwen38/bootstrap.py", line 447, in _prepare_model
    operation = PreparedMixedRound(mixed_owner, batch, width, prefill, name,
  File "ignition/recipes/qwen38/mixed_model.py", line 87, in __init__
    self.round = PreparedSpeculativeRound(decode_owner, batch, width, name,
  File "ignition/recipes/qwen38/speculative_model.py", line 166, in __init__
    self.decoder = decoder_factory(owner.weights, scratch, self.state, owner.cache, owner.rotary,
  File "ignition/recipes/qwen38/mixed_model.py", line 85, in speculative_decoder
    view = decoder_factory(*args, **kwargs)
  File "ignition/recipes/qwen38/mixed_model.py", line 69, in decoder_factory
    self.decoder = PreparedDecoder(weights, full_scratch, state, cache, rotary, table, workspaces,
  File "ignition/recipes/qwen38/decoder.py", line 223, in __init__
    raise ValueError("verification requires one sample row per hidden row")
ValueError: verification requires one sample row per hidden row

317c1ecc added this check to PreparedDecoder and aliases sample_hidden to hidden when verification=True (decoder.py#L221-L224). The mixed round's decoder packs the prompt and decode tokens into self.packing.tokens hidden rows. It samples only batch * width + 1 of them, the decode rows plus the prompt's last row, and passes verification=True (mixed_model.py#L65-L70). So the shapes never match for a mixed graph. At ca1f960e the graph catalog completes (159/159).

2. speculative graph lacks explicit proposal depth (since bce032c0)

At ca1f960e, every preparation phase passes: graph catalog, kernel tuning, compile, warm-up and graph capture. Then:

[  FAILED  ] Export native package.  8.9s
  File "ignition/preparation/runtime_package.py", line 134, in _capture
    self.native_output = flare_heliopause.export_capture(json.dumps(request), owners, recorder)
RuntimeError: speculative graph lacks explicit proposal depth
error: native package export failed: speculative graph lacks explicit proposal depth

The two sides disagree about mixed graphs:

  • Heliopause requires draft_tokens when kind == "speculative" || kind == "mixed" && speculative (binding.rs#L188-L194).
  • Ignition never provides one for a mixed graph:

Before bce032c0, binding.rs filled in draft_tokens = width - 1 for these graphs, and the loader still expects that. For a mixed graph with speculation enabled, selection.rs:

Dropping the mixed case from the exporter check is not enough. The export then succeeds, but flare run fails at "Prepare model" with invalid or duplicate serving graph variant (selection.rs#L236).

Workaround I tested

Restore the earlier default in the exporter, for mixed graphs only:

--- a/heliopause/src/export/binding.rs
+++ b/heliopause/src/export/binding.rs
@@ -187,7 +187,11 @@ pub(crate) fn execution(
             .unwrap_or(false);
         if kind == "speculative" || kind == "mixed" && speculative {
             if !graph.contains_key("draft_tokens") {
-                return Err(super::invalid("speculative graph lacks explicit proposal depth"));
+                // Mixed graphs carry no explicit depth from preparation; use the pre-ABI4 default.
+                if kind != "mixed" {
+                    return Err(super::invalid("speculative graph lacks explicit proposal depth"));
+                }
+                graph.insert("draft_tokens".into(), width.saturating_sub(1).into());
             }
             graph.insert("verify_tokens".into(), width.into());
             graph.insert("draft_source".into(), "MTP".into());

Results at ca1f960e with this change:

  • Export and load: the export completes and the package loads: 76.5 GiB GPU, 1.79M KV tokens.
  • Serving: flare run … --no-mtp-adaptive-verification --mtp-draft-tokens 3 served without errors through:
    • two runs of 8 concurrent multi-turn sessions (48 turns each), with flare_graph_launches_total{kind="mixed"} at 24 and 53;
    • single-request decode and 32K-token prompt probes.

A cleaner fix may belong on the Ignition side: let a mixed graph with speculation declare its depth (width - 1 in the Qwen recipe), so the exporter check can stay strict.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.