Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

perf(moe): prefetch the next layer's experts from a router lookahead during offloaded decode

Aperta
  • Qualcosa non va?
#677 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
18/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, performance

Direzione di ricerca

Start from the offloaded decode path behind --moe-strategy offload and the expert slot-map ensure logic, then read the linked fork commits and PR #74 to see the proposed router lookahead and prefetch install. Any change must keep every slot-map write on the compute stream, so review the race tests before anything else. Done means the FREETOKEN_MOE_PREFETCH=on path gives token-identical greedy output on cold prompts with the slot-map audit enabled.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

In offloaded decode (--moe-strategy offload, decode on the GPU), each layer's expert misses are copied over PCIe only after that layer's router has run, so the copy sits on the critical path. On an RTX 5080 (16 GB, PCIe) serving Qwen3.8-Flash-Next-NVFP4, the expert H2D copies are about half of each decode step.

Proposal: predict layer L's experts one layer early and copy them while layer L-1 computes.

  • Lookahead: run layer L's router on layer L-1's router input (the residual stream changes little between adjacent layers). Run it on a side stream forked at the start of L-1's MoE block.
  • Install: right after L-1's own ensure, install up to N predicted, non-resident experts as low-priority slots: held, usage 0, first to be evicted. Copy them on a dedicated high-priority stream behind L-1's demand copy. L's demand copy then waits on that stream.
  • Correctness constraint: every slot-map writer (demand ensure and prefetch install) must stay on the compute stream. Only reads and byte copies may cross streams. A first version that installed from the predictor stream behind an event join corrupted 1–6 of 60 cold greedy prompts. Two writers racing on the slot map was enough, even though an nsys trace showed no ordering violation.

Measured (Qwen3.8-Flash-Next-NVFP4, scored eviction with the shared-expert overlap as the baseline, N = 4):

baseline with prefetch
decode, 32K context 43.4 tok/s 53.0 tok/s (+22%)
decode, 255K context 43.0 tok/s 50.8 tok/s
demand misses per token 206 111
VRAM +148 MiB

The output is lossless: greedy outputs were token-identical on 420 cold nonce prompts, with a slot-map audit enabled.

Implementation (on my fork, against an older main):

  • 8c171b6: router lookahead (layer L's router on layer L-1's input).
  • 47d4811: prefetch into low-priority slots behind FREETOKEN_MOE_PREFETCH=on.
  • 3a0ff77: install prefetches on the compute stream, so the slot map has a single writer.
  • The full series, including the race tests and the slot-audit/verify debug modes: https://github.com/gdevenyi/FreeToken/pull/74
Lingua principale
Python
Stelle
14.2k
Fork
1.4k
Merge medio
21h 53m
PR unite (30g)
100

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di FlashML-org/FreeToken

Tutte le issue di FlashML-org/FreeToken

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.