Misc. bug: DFlash profit draft-max controller never leaves max depth (exploration not wired since v0.4.0 rebase, biased cycle timings)
Personne n'a encore pris cette issue.
Évaluation
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Accessibilité débutants
- 18/100
- Type d'issue
- Bug
- Clarté
- Plutôt claire
- Activité
- Active
- Domaine
- backend, performance
Piste de recherche
The problem sits in tools/server/server-context.cpp (server_slot::get_n_draft_max() near line 719, and the DONE_PROMPT -> GENERATING transition near line 5527) and in server-adaptive-dm.h, where the unused profit_* probe helpers live. The first step is to reproduce the stuck draft-max with --spec-dm-controller profit and the traces described. Done would be a fix for the first-token baseline timing and a controller that can move off depth 15, backed by a test; the CUDA graph key in ggml/src/ggml-cuda/ggml-cuda.cu is a separate, deeper change.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Description
Name and Version
version: 0.4.9-dev (build 12930, commit b46c53e54)
built with MSVC 19.51.36256.0 for Windows AMD64
CUDA build, CMAKE_CUDA_ARCHITECTURES=120, GGML_CUDA_FA=ON.
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server -m Qwen3.5-4B-Q8_0.gguf -md qwen35-4b-dflash-Q8_0.gguf --spec-type draft-dflash -ngl 99 -c 65536 -b 512 -ub 512 -fa on -np 1 --spec-dm-controller profit --load-mode none -lv 4 --reasoning-loop-guard off
Target unsloth/Qwen3.5-4B-GGUF Q8_0, drafter Anbeeld/Qwen3.5-4B-DFlash-GGUF Q8_0, RTX 5060 Ti 16 GB (sm_120),
PCIe 3.0 x8.
Problem description & steps to reproduce
With --spec-dm-controller profit, DFlash1 holds draft-max 15 for the whole session. It does this even when
speculation is clearly a loss. Three separate causes were found:
1. Depth exploration is no longer scheduled.
server_slot::get_n_draft_max() (tools/server/server-context.cpp:719) only returns adaptive_n_max. The excursions that get_dflash_n_draft_max() used to schedule (4039a8ffd, 57da314b5) were dropped in the v0.4.0 rebase c9e746733, together with tests/test-dflash-plumbing.cpp. The helpers are still in server-adaptive-dm.h, but nothing calls them: profit_next_unready_probe_depth, profit_next_unready_explore_depth, profit_should_probe_baseline, profit_mark_baseline_probe,
profit_next_off_probe_depth.
As a result:
- after the cold-start baseline, no depth other than the held one is ever measured;
- periodic baseline probes never happen;
- the off state has no reprobe.
The controller can still step down through the survival estimate, but only when the held depth is clearly worse than the baseline (hold_episode_clearly_bad).
Claude re-wired the helpers in get_n_draft_max() (baseline probe, off-state reprobe, an excursion every 12 cycles). The controller then does measure other depths, but it still never switches. The next two findings explain why.
2. The first token of every request is recorded as a no-spec baseline cycle.
adaptive_cycle_start_us is only set for slots that are already generating (server-context.cpp:4260). The token sampled right after SLOT_STATE_DONE_PROMPT -> GENERATING (:5527) goes through the no-spec timing path (:5558). Its cycle time is measured from the last cycle of the previous request, so it includes the idle gap and prompt processing. With temporary tracing, these samples are 143–164 ms, while normal baseline cycles are 21–34 ms. profit_score_for_stats() works on totals and there are only a few baseline samples (3–8 here), so one such sample pulls the baseline score from ~38 down to 13–15 tok/s (see log). Likely fix (not built or measured): set adaptive_cycle_start_us = 0 on the DONE_PROMPT -> GENERATING transition.
3. Sporadic cycles at a different depth run without CUDA graphs.
ggml_cuda_graph_get_key() (ggml/src/ggml-cuda/ggml-cuda.cu:2846) returns cgraph->nodes[0], which is the same for every verify batch size. Each shape change resets warmup_complete, so a single excursion or baseline cycle runs eagerly. Mean cycle time per requested depth (traced, 8k):
| cycle | graphs on | GGML_CUDA_DISABLE_GRAPHS=1 |
key incl. nodes[0]->ne (experiment) |
|---|---|---|---|
| depth 15, steady (n≈900) | 28.6 ms | 36.3 | 27.9 |
| depths 1–13 (3–27 samples each) | 35–38 | 36–39 | 31–37 |
| baseline depth 0 (n=12) | 28.0 | 30.0 | 27.3 |
Excursions therefore look ~30 % more expensive than they would be at steady state, and a shallower depth can practically never win. A per-shape graph key alone does not fix this, because a single cycle never gets past warmup and capture. Excursions would probably have to last several cycles, with the first samples discarded.
Measurements. The prompt is the first N tokens of a large Python file (django.py). In the greedy runs each segment continues the original text (prompt = N + i·384 tokens, 5×384 generated), so every variant sees identical inputs. "orig" = main b46c53e54; "explore" = main with the exploration re-wired (finding 1 only).
A. Greedy code, tok/s:
| ctx | orig | explore | static n15 | static n7 | static n3 |
|---|---|---|---|---|---|
| 8k | 171.1 | 162.4 | 170.5 | 175.4 | 146.5 |
| 60k | 318.4 | 310.2 | 317.6 | 245.6 | 146.7 |
Both controllers stay at 15. Exploration costs 2–5 % and never pays off here.
B. Drafter-hostile text (temperature 2.0, top-k 40, 4×256 tokens), then greedy code (4×384), 8k, tok/s:
| phase | orig | explore | no drafter | DFlash loaded, --spec-draft-n-max 0 |
|---|---|---|---|---|
| hostile | 45.8 | 44.1 | 73.1 | – |
| code | 169.0 | 142.8 | 73.0 | 56 (greedy code) |
Acceptance in the hostile phase is ~2 %. Both controllers stay at 15 there. Inside DFlash the best choice would be
off (≈56 tok/s), not 45.
For scale: on this code, a perfect choice would gain about +3 % at 8k (n7 over n15) and nothing at 60k. In drafter-hostile phases it would gain about +24 %. Even with draft depth 0, a loaded DFlash drafter costs ~4 ms per token (56 vs 73 tok/s).
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
Side note: the issue templates still link to the AI policy of ggml-org/llama.cpp, while this repository's own CONTRIBUTING.md allows AI-assisted issues with disclosure.
First Bad Commit
c9e746733 (Complete BeeLlama v0.4.0 upstream rebase) for finding 1. Findings 2 and 3 were not bisected.
Relevant log output
Logs
Controller on main (orig), run B: warm-up request (cold start), first hostile request, and first code request
(new sampling settings, so the stats are reset). These are the only transitions in the whole run. score is the
baseline score in tok/s; the real no-drafter rate is 73 tok/s.
slot apply_adapti: id 0 | task 0 | adaptive draft-max profit: 0 -> 15 (score=38.03)
slot apply_adapti: id 0 | task 32 | adaptive draft-max profit: 0 -> 15 (score=14.98)
slot apply_adapti: id 0 | task 836 | adaptive draft-max profit: 0 -> 15 (score=15.15)
Temporary tracing (LLAMA_DM_TRACE, not part of main) with exploration re-wired:
DM-TRACE baseline cycle_ms=30.21 first_after_prompt=0 baseline_score=36.75 samples=4 n_max=15
DM-TRACE baseline cycle_ms=161.98 first_after_prompt=1 baseline_score=0.00 samples=1 n_max=0
DM-TRACE baseline cycle_ms=29.82 first_after_prompt=0 baseline_score=0.00 samples=2 n_max=0
DM-TRACE spec req=12 n_draft=12 acc=3 draft_ms=5.73 verify_ms=1.55 cycle_ms=45.84 n_max=15
DM-TRACE spec req=8 n_draft=8 acc=2 draft_ms=5.34 verify_ms=1.53 cycle_ms=37.40 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=1 draft_ms=5.33 verify_ms=1.18 cycle_ms=27.62 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=0 draft_ms=5.51 verify_ms=0.79 cycle_ms=26.84 n_max=15
- Langage dominant
- C++
- Étoiles
- 1.2k
- Forks
- 80
- Merge moyen
- 4 j 20 h
- PR mergées (30 j)
- 2
Préparer son environnement
Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.
Par où commencer
- Lisez l'issue en entier, puis le guide de contribution du projet.
- Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
- Forkez le dépôt et travaillez sur une branche.
- Ouvrez une pull request qui référence le numéro de l'issue.
Autres issues de Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotOuverte
Difficulté 2/5 1-3 heures Accessibilité débutants 68/100
Anbeeld/beellama.cpp#205 ·
-
Difficulté 3/5 Une demi-journée Accessibilité débutants 58/100
Anbeeld/beellama.cpp#206 ·
-
Difficulté 3/5 1-2 jours Accessibilité débutants 57/100
Anbeeld/beellama.cpp#204 ·
-
Difficulté 4/5 3-5 jours Accessibilité débutants 45/100
Anbeeld/beellama.cpp#203 · 2 commentaires ·
-
Difficulté 4/5 3-5 jours Accessibilité débutants 45/100
Anbeeld/beellama.cpp#202 · 1 commentaire ·
Toutes les issues de Anbeeld/beellama.cpp
Issues similaires
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
CachyOS/packageinstaller#144 ·
Les mainteneurs répondent en général sous 1 jour
-
documentation
Difficulté 1/5 Moins d'une heure Accessibilité débutants 82/100
TomerAberbach/profiler-md#951 ·
Les mainteneurs répondent en général sous 1 jour
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPeut-être pris @0z5a l’a pris aujourd’hui. Ouverte
Difficulté 2/5 1-3 heures Accessibilité débutants 82/100
apache/pulsar-client-go#1575 ·
Les mainteneurs répondent en général sous 2 jours
-
Difficulté 2/5 1-3 heures Accessibilité débutants 72/100
pocoproject/poco#5764 ·
Les mainteneurs répondent en général sous 2 jours
-
bug cpp-core-guidelines low-priority
Difficulté 2/5 1-3 heures Accessibilité débutants 62/100
NatronGitHub/openfx-supportext#164 ·
Les mainteneurs répondent en général sous 1 jour