Misc. bug: DFlash profit draft-max controller never leaves max depth (exploration not wired since v0.4.0 rebase, biased cycle timings)

Aperta
#195 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, python

Direzione di ricerca

The problem sits in tools/server/server-context.cpp (server_slot::get_n_draft_max() near line 719, and the DONE_PROMPT -> GENERATING transition near line 5527) and in server-adaptive-dm.h, where the unused profit_* probe helpers live. The first step is to reproduce the stuck draft-max with --spec-dm-controller profit and the traces described. Done would be a fix for the first-token baseline timing and a controller that can move off depth 15, backed by a test; the CUDA graph key in ggml/src/ggml-cuda/ggml-cuda.cu is a separate, deeper change.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Name and Version
version: 0.4.9-dev (build 12930, commit b46c53e54)
built with MSVC 19.51.36256.0 for Windows AMD64

CUDA build, CMAKE_CUDA_ARCHITECTURES=120, GGML_CUDA_FA=ON.

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line
llama-server -m Qwen3.5-4B-Q8_0.gguf -md qwen35-4b-dflash-Q8_0.gguf --spec-type draft-dflash -ngl 99 -c 65536 -b 512 -ub 512 -fa on -np 1 --spec-dm-controller profit --load-mode none -lv 4 --reasoning-loop-guard off

Target unsloth/Qwen3.5-4B-GGUF Q8_0, drafter Anbeeld/Qwen3.5-4B-DFlash-GGUF Q8_0, RTX 5060 Ti 16 GB (sm_120),
PCIe 3.0 x8.

Problem description & steps to reproduce

With --spec-dm-controller profit, DFlash1 holds draft-max 15 for the whole session. It does this even when
speculation is clearly a loss. Three separate causes were found:

1. Depth exploration is no longer scheduled.
server_slot::get_n_draft_max() (tools/server/server-context.cpp:719) only returns adaptive_n_max. The excursions that get_dflash_n_draft_max() used to schedule (4039a8ffd, 57da314b5) were dropped in the v0.4.0 rebase c9e746733, together with tests/test-dflash-plumbing.cpp. The helpers are still in server-adaptive-dm.h, but nothing calls them: profit_next_unready_probe_depth, profit_next_unready_explore_depth, profit_should_probe_baseline, profit_mark_baseline_probe,
profit_next_off_probe_depth.
As a result:

  • after the cold-start baseline, no depth other than the held one is ever measured;
  • periodic baseline probes never happen;
  • the off state has no reprobe.

The controller can still step down through the survival estimate, but only when the held depth is clearly worse than the baseline (hold_episode_clearly_bad).

Claude re-wired the helpers in get_n_draft_max() (baseline probe, off-state reprobe, an excursion every 12 cycles). The controller then does measure other depths, but it still never switches. The next two findings explain why.

2. The first token of every request is recorded as a no-spec baseline cycle.
adaptive_cycle_start_us is only set for slots that are already generating (server-context.cpp:4260). The token sampled right after SLOT_STATE_DONE_PROMPT -> GENERATING (:5527) goes through the no-spec timing path (:5558). Its cycle time is measured from the last cycle of the previous request, so it includes the idle gap and prompt processing. With temporary tracing, these samples are 143–164 ms, while normal baseline cycles are 21–34 ms. profit_score_for_stats() works on totals and there are only a few baseline samples (3–8 here), so one such sample pulls the baseline score from ~38 down to 13–15 tok/s (see log). Likely fix (not built or measured): set adaptive_cycle_start_us = 0 on the DONE_PROMPT -> GENERATING transition.

3. Sporadic cycles at a different depth run without CUDA graphs.
ggml_cuda_graph_get_key() (ggml/src/ggml-cuda/ggml-cuda.cu:2846) returns cgraph->nodes[0], which is the same for every verify batch size. Each shape change resets warmup_complete, so a single excursion or baseline cycle runs eagerly. Mean cycle time per requested depth (traced, 8k):

cycle graphs on GGML_CUDA_DISABLE_GRAPHS=1 key incl. nodes[0]->ne (experiment)
depth 15, steady (n≈900) 28.6 ms 36.3 27.9
depths 1–13 (3–27 samples each) 35–38 36–39 31–37
baseline depth 0 (n=12) 28.0 30.0 27.3

Excursions therefore look ~30 % more expensive than they would be at steady state, and a shallower depth can practically never win. A per-shape graph key alone does not fix this, because a single cycle never gets past warmup and capture. Excursions would probably have to last several cycles, with the first samples discarded.

Measurements. The prompt is the first N tokens of a large Python file (django.py). In the greedy runs each segment continues the original text (prompt = N + i·384 tokens, 5×384 generated), so every variant sees identical inputs. "orig" = main b46c53e54; "explore" = main with the exploration re-wired (finding 1 only).

A. Greedy code, tok/s:

ctx orig explore static n15 static n7 static n3
8k 171.1 162.4 170.5 175.4 146.5
60k 318.4 310.2 317.6 245.6 146.7

Both controllers stay at 15. Exploration costs 2–5 % and never pays off here.

B. Drafter-hostile text (temperature 2.0, top-k 40, 4×256 tokens), then greedy code (4×384), 8k, tok/s:

phase orig explore no drafter DFlash loaded, --spec-draft-n-max 0
hostile 45.8 44.1 73.1 –
code 169.0 142.8 73.0 56 (greedy code)

Acceptance in the hostile phase is ~2 %. Both controllers stay at 15 there. Inside DFlash the best choice would be
off (≈56 tok/s), not 45.

For scale: on this code, a perfect choice would gain about +3 % at 8k (n7 over n15) and nothing at 60k. In drafter-hostile phases it would gain about +24 %. Even with draft depth 0, a loaded DFlash drafter costs ~4 ms per token (56 vs 73 tok/s).

AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.

Side note: the issue templates still link to the AI policy of ggml-org/llama.cpp, while this repository's own CONTRIBUTING.md allows AI-assisted issues with disclosure.

First Bad Commit

c9e746733 (Complete BeeLlama v0.4.0 upstream rebase) for finding 1. Findings 2 and 3 were not bisected.

Relevant log output
Logs

Controller on main (orig), run B: warm-up request (cold start), first hostile request, and first code request
(new sampling settings, so the stats are reset). These are the only transitions in the whole run. score is the
baseline score in tok/s; the real no-drafter rate is 73 tok/s.

slot apply_adapti: id  0 | task 0 | adaptive draft-max profit: 0 -> 15 (score=38.03)
slot apply_adapti: id  0 | task 32 | adaptive draft-max profit: 0 -> 15 (score=14.98)
slot apply_adapti: id  0 | task 836 | adaptive draft-max profit: 0 -> 15 (score=15.15)

Temporary tracing (LLAMA_DM_TRACE, not part of main) with exploration re-wired:

DM-TRACE baseline cycle_ms=30.21 first_after_prompt=0 baseline_score=36.75 samples=4 n_max=15
DM-TRACE baseline cycle_ms=161.98 first_after_prompt=1 baseline_score=0.00 samples=1 n_max=0
DM-TRACE baseline cycle_ms=29.82 first_after_prompt=0 baseline_score=0.00 samples=2 n_max=0
DM-TRACE spec req=12 n_draft=12 acc=3 draft_ms=5.73 verify_ms=1.55 cycle_ms=45.84 n_max=15
DM-TRACE spec req=8 n_draft=8 acc=2 draft_ms=5.34 verify_ms=1.53 cycle_ms=37.40 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=1 draft_ms=5.33 verify_ms=1.18 cycle_ms=27.62 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=0 draft_ms=5.51 verify_ms=0.79 cycle_ms=26.84 n_max=15
Lingua principale
C++
Stelle
1.2k
Fork
80
Merge medio
4g 20h
PR unite (30g)
2

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Anbeeld/beellama.cpp

Tutte le issue di Anbeeld/beellama.cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.