DeepSeek-V4-Flash-Vision-Exp TP2/TP4: b12x tiny-decode MoE kernel skips `swiglu_limit` (fix in local-inference-lab/flashinfer#83)

Abierto
  • ¿Algo no está bien?
#249 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
rust
Área
performance

Línea de trabajo

The issue identifies the affected FlashInfer tiny-decode kernel and the proposed fix in local-inference-lab/flashinfer#83. Read that pull request and inspect the kernel path for 1–4 token MoE launches, then check the reported TP2 and TP4 shard shapes and GPU test. Done means the tiny kernel applies swiglu_limit and calls without a limit retain current behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

For MoE launches of 1-4 tokens, b12x's W4A8-MX path can select the tiny-decode kernel. That kernel computes plain SiLU-GLU and never applies swiglu_limit. The deepseek4 recipe's checkpoint (DeepSeek-V4-Flash-Vision-Exp-lossless-CSF) sets swiglu_limit = 10.0, and its TP2 and TP4 expert shards are tiny-decode eligible. So 1-4 token decode steps (one to four active sequences, small verification rounds, draft MoE) run the routed experts without the clamp whenever a gate/up activation exceeds 10.

The fix is open as local-inference-lab/flashinfer#83: the tiny kernel applies the clamp, and calls without a limit compile to today's code.

Affected (Hyperflare main, FlashInfer pin local-inference-lab/flashinfer@e0f105b3)

Model (recipe) TP Per-rank expert width (hidden) 1-4 token kernel Limit applied
DeepSeek-V4-Flash-Vision-Exp (deepseek4) 2 1024 (4096) tiny no
4 512 (4096) tiny no
DeepSeek-V4.1-Flash (deepseek41) 4 576 (5120) compact micro yes

NVFP4 recipes (GLM-5.3-Flash, MiniMax-M3, Step5) don't reach this kernel.

Evidence (RTX PRO 6000 Blackwell, SM120)

  • Kernel level, at the DeepSeek-V4 TP2 and TP4 shard shapes (k4096/n1024 and k4096/n512, 256 experts, top-6, 1 and 4 tokens; the GPU test in #83):

    • on main, selection picks tiny and the launch doesn't receive the limit;
    • with inputs that exceed the limit, the output misses the clamped reference;
    • with #83, it matches a tiny-arithmetic oracle (cos ≥ 0.9999, normalized RMSE ≤ 1%).

  • End to end on the shipped config (Hyperflare main 8c2e782, DeepSeek-V4-Flash-Vision-Exp TP2 on 2x RTX PRO 6000, --no-mtp so every decode step is a 1-row MoE launch; decode vs teacher-forced prefill; two separate exports per variant, 20 + 40 runs = 18,000 tokens each):

    FlashInfer Confidently wrong tokens (decode drop, prefill top-1 > 0.74) Sudden decode drops (top-1 < 0.2 after two confident tokens), prefill also unsure
    pin e0f105b3 (tiny kernel, no clamp) 0 327 (89 + 238), 1.8% of tokens
    pin + #83 clamp 0 63 (14 + 49), 0.35% of tokens

    On DS4 the missing clamp shows as ~5x more sudden low-confidence decode steps (6.4x and 4.9x in the two runs) rather than confidently wrong tokens.

  • End to end on DeepSeek-V4.1-Flash at TP3 (a downstream, non-shipped config, same tiny shape class): 10 of 20 runs emitted a token that teacher-forced prefill rejects with high confidence; with the clamp, 0 of 20. Replay of captured bad steps: routed MoE output 9-18x off unclamped, 0.4-0.5% from an exact float64 recomputation with the clamp.

Note: exporting deepseek4 on main currently fails before this point: ignition/components/weights.py lists an engram field for ignition.recipes.deepseek4.weights.LayerWeights, which has none ('LayerWeights' object has no attribute 'engram' in rebuild_manifest.declared_views). The measurement above dropped engram from that entry; nothing else was changed.

Workaround until #83 lands

B12X_W4A8_TINY_DECODE=0 keeps the tiny kernel off (+34% per MoE call at 1 token). Prepared packages and tuning made with tiny enabled must be re-prepared, because the switch isn't part of the tuning key. The deepseek4 recipe could also set it through b12x_environment.

Lenguaje dominante
Rust
Estrellas
800
Forks
106
Merge medio
1 d 3 h
PR fusionados (30 d)
100

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/hyperflare-inference

Todos los issues de NVIDIA/hyperflare-inference

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.