Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Engine] MTP drafts never proposed on sm_89 (RTX 4060 Ti 16GB, hybrid mode, drafts accepted 0 of 0)

Aperta
  • Qualcosa non va?
#4,575 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
43/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, performance

Direzione di ricerca

Start by reading docs/QWEN36.md and trace the engine path that loads the MTP block and reports draft acceptance; the issue does not name source files or tests for that path. Reproduce the Windows sm_89 hybrid setup with the provided launch command and request, then inspect proposer initialization and diagnostics. Done means establishing why drafts remain 0 of 0 and documenting or fixing the behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

MTP speculative decoding never produces a single draft on an RTX 4060 Ti 16 GB (sm_89) / Windows 10, so decode speed stays well below the reference numbers for this model. The MTP block is found and loaded, --mtp 2 is passed to the engine, but every session reports drafts accepted 0 of 0.

This may be relevant because docs/QWEN36.md states the sm_89 build was compiled in CI but never actually executed on real hardware.

Environment

Item Value
GPU NVIDIA GeForce RTX 4060 Ti 16 GB (compute capability 8.9 / sm_89)
Driver 610.88
CPU AMD Ryzen 7 5700X3D (8C / 16T)
RAM 32 GB DDR4-3200 CL18, dual channel (2×16 GB)
OS Windows 10 Pro 22H2, build 19045
Strata Terminal 0.2.7 (build 0.1.53, python 3.12.15)
Engine 0.1.52-windows-x86_64-cuda-cuda13
Model unsloth/Qwen3.6-35B-A3B-MTP-GGUF → Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (21.28 GiB), downloaded via --source modelscope

Reproduction

strata launch --family qwen36 --model ud-q4_k_xl --context 32k --yes --no-tui

Then a plain OpenAI-compatible request (400 generated tokens):

curl -s --noproxy '*' http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6-35b-a3b-ud-q4_k_xl","messages":[{"role":"user","content":"Write a detailed essay about the history of tea."}],"max_tokens":400,"temperature":0}'

Observed behaviour

Engine is launched with MTP explicitly requested, and the log confirms the MTP block was loaded:

strata.exe --serve --native ...\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --max-context 32768 --prefill auto --mtp 2
strata-q36: 753 tensors (layout checked), 4.40 GiB on the GPU in 4.9 s
strata-q36: prompt experts on MMQ (int8)
strata-q36: context 32768 cells, prompt chunk 2048, MTP block loaded, 6.69 GiB of VRAM free
strata-q36: hybrid experts: 66 of 256 per layer in VRAM (4.83 GiB), all 18.75 GiB in pageable host memory,
            the rest of a batch of <= 128 rows on 8 CPU threads, bigger prompt chunks streamed (1.10 GiB staging)

But on every single request, including long 400-token generations:

strata serve: session 0: prompt 33 tokens = 0 reused + 33 read in 980 ms (33.7 tok/s),
              400 generated in 14135 ms (28.3 tok/s), drafts accepted 0 of 0 (length)
strata serve: session 0: prompt 11 tokens = 0 reused + 11 read in 625 ms (17.6 tok/s),
              1 generated in 2 ms (462.1 tok/s), drafts accepted 0 of 0 (length)

0 of 0 (not 0 of N) looks like drafts are never proposed, rather than proposed and rejected.

Measured speed

Config Decode Prompt read
DDR4-2400 (A-XMP off) 40.0 / 40.7 tok/s 103 tok/s
DDR4-3200 (A-XMP on) 45.7 / 46.1 tok/s 177 tok/s
8K context instead of 32K 31.0 / 43.2 tok/s —

Reference points from the docs for the same model and quant: R9700 32 GB = 117.8 tok/s, RTX 3060 12 GB (hybrid, only 37/256 experts in VRAM) = 58.5–59.7 tok/s. This machine places more experts in VRAM (66/256) than the 3060 and has higher VRAM bandwidth, yet is ~20% slower — consistent with MTP being inactive while the 3060 numbers presumably include it.

What I already tried (none changed 0 of 0)

  1. --tune quick — reports already fastest, no parameter change.
  2. --context 8k instead of 32k (frees VRAM for expert cache) — slightly faster, still 0 of 0.
  3. Explicit --mtp on / --set mtp=... variants — no drafts.
  4. --set pool-workers=16 --set pool-affinity=all — 35% slower (30 tok/s). The engine correctly warns: "16 CPU expert workers are configured, but this process can run only 16 CPU threads. The extra workers spin... Recommended: --pool-workers 7". Reverted to default 8.

Worker scaling, measured on the same box after enabling A-XMP (400 generated tokens, engine-reported tok/s):

--pool-workers Decode
6 (7 CPU threads) 38.7 / 46.0 / 45.4
8 (default) 45.7 / 46.1
16 (all logical threads) 30

6 and 8 are identical within noise, and 16 is much worse. So the CPU expert path looks bandwidth-saturated at ~6 threads rather than compute-bound — which would also explain why going from DDR4-2400 to DDR4-3200 (+33% memory bandwidth) only bought +13% decode. Worth noting in case it informs default worker heuristics: the engine picks physical-core count (8), but 6 already saturates DDR4 dual channel here.

Questions

  1. Is MTP draft generation expected to work when the model runs in hybrid mode with all 18.75 GiB in pageable host memory? I could not find a documented restriction, and MTP block loaded suggests it should be active.
  2. Does the draft model require additional VRAM beyond the 6.69 GiB reported free? If so, the log could say so instead of silently producing zero drafts.
  3. Is there a way to get more verbose MTP diagnostics (e.g. draft proposer init/failure reason)? I did not find a flag for it.

Happy to run any diagnostic build or provide full logs / traces — the machine is available and reproducible. I can also re-test on other quant levels (UD-Q6_K_XL needs 48 GB RAM, which I can add) to see whether the behaviour is quant-specific.

Lingua principale
C++
Stelle
11.6k
Fork
1k
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Niko1221/Strata

Tutte le issue di Niko1221/Strata

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.