Misc. bug: DFlash profit draft-max controller never leaves max depth (exploration not wired since v0.4.0 rebase, biased cycle timings)
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 18/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Activo
- Área
- backend, performance
Línea de trabajo
The problem sits in tools/server/server-context.cpp (server_slot::get_n_draft_max() near line 719, and the DONE_PROMPT -> GENERATING transition near line 5527) and in server-adaptive-dm.h, where the unused profit_* probe helpers live. The first step is to reproduce the stuck draft-max with --spec-dm-controller profit and the traces described. Done would be a fix for the first-token baseline timing and a controller that can move off depth 15, backed by a test; the CUDA graph key in ggml/src/ggml-cuda/ggml-cuda.cu is a separate, deeper change.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Name and Version
version: 0.4.9-dev (build 12930, commit b46c53e54)
built with MSVC 19.51.36256.0 for Windows AMD64
CUDA build, CMAKE_CUDA_ARCHITECTURES=120, GGML_CUDA_FA=ON.
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server -m Qwen3.5-4B-Q8_0.gguf -md qwen35-4b-dflash-Q8_0.gguf --spec-type draft-dflash -ngl 99 -c 65536 -b 512 -ub 512 -fa on -np 1 --spec-dm-controller profit --load-mode none -lv 4 --reasoning-loop-guard off
Target unsloth/Qwen3.5-4B-GGUF Q8_0, drafter Anbeeld/Qwen3.5-4B-DFlash-GGUF Q8_0, RTX 5060 Ti 16 GB (sm_120),
PCIe 3.0 x8.
Problem description & steps to reproduce
With --spec-dm-controller profit, DFlash1 holds draft-max 15 for the whole session. It does this even when
speculation is clearly a loss. Three separate causes were found:
1. Depth exploration is no longer scheduled.
server_slot::get_n_draft_max() (tools/server/server-context.cpp:719) only returns adaptive_n_max. The excursions that get_dflash_n_draft_max() used to schedule (4039a8ffd, 57da314b5) were dropped in the v0.4.0 rebase c9e746733, together with tests/test-dflash-plumbing.cpp. The helpers are still in server-adaptive-dm.h, but nothing calls them: profit_next_unready_probe_depth, profit_next_unready_explore_depth, profit_should_probe_baseline, profit_mark_baseline_probe,
profit_next_off_probe_depth.
As a result:
- after the cold-start baseline, no depth other than the held one is ever measured;
- periodic baseline probes never happen;
- the off state has no reprobe.
The controller can still step down through the survival estimate, but only when the held depth is clearly worse than the baseline (hold_episode_clearly_bad).
Claude re-wired the helpers in get_n_draft_max() (baseline probe, off-state reprobe, an excursion every 12 cycles). The controller then does measure other depths, but it still never switches. The next two findings explain why.
2. The first token of every request is recorded as a no-spec baseline cycle.
adaptive_cycle_start_us is only set for slots that are already generating (server-context.cpp:4260). The token sampled right after SLOT_STATE_DONE_PROMPT -> GENERATING (:5527) goes through the no-spec timing path (:5558). Its cycle time is measured from the last cycle of the previous request, so it includes the idle gap and prompt processing. With temporary tracing, these samples are 143–164 ms, while normal baseline cycles are 21–34 ms. profit_score_for_stats() works on totals and there are only a few baseline samples (3–8 here), so one such sample pulls the baseline score from ~38 down to 13–15 tok/s (see log). Likely fix (not built or measured): set adaptive_cycle_start_us = 0 on the DONE_PROMPT -> GENERATING transition.
3. Sporadic cycles at a different depth run without CUDA graphs.
ggml_cuda_graph_get_key() (ggml/src/ggml-cuda/ggml-cuda.cu:2846) returns cgraph->nodes[0], which is the same for every verify batch size. Each shape change resets warmup_complete, so a single excursion or baseline cycle runs eagerly. Mean cycle time per requested depth (traced, 8k):
| cycle | graphs on | GGML_CUDA_DISABLE_GRAPHS=1 |
key incl. nodes[0]->ne (experiment) |
|---|---|---|---|
| depth 15, steady (n≈900) | 28.6 ms | 36.3 | 27.9 |
| depths 1–13 (3–27 samples each) | 35–38 | 36–39 | 31–37 |
| baseline depth 0 (n=12) | 28.0 | 30.0 | 27.3 |
Excursions therefore look ~30 % more expensive than they would be at steady state, and a shallower depth can practically never win. A per-shape graph key alone does not fix this, because a single cycle never gets past warmup and capture. Excursions would probably have to last several cycles, with the first samples discarded.
Measurements. The prompt is the first N tokens of a large Python file (django.py). In the greedy runs each segment continues the original text (prompt = N + i·384 tokens, 5×384 generated), so every variant sees identical inputs. "orig" = main b46c53e54; "explore" = main with the exploration re-wired (finding 1 only).
A. Greedy code, tok/s:
| ctx | orig | explore | static n15 | static n7 | static n3 |
|---|---|---|---|---|---|
| 8k | 171.1 | 162.4 | 170.5 | 175.4 | 146.5 |
| 60k | 318.4 | 310.2 | 317.6 | 245.6 | 146.7 |
Both controllers stay at 15. Exploration costs 2–5 % and never pays off here.
B. Drafter-hostile text (temperature 2.0, top-k 40, 4×256 tokens), then greedy code (4×384), 8k, tok/s:
| phase | orig | explore | no drafter | DFlash loaded, --spec-draft-n-max 0 |
|---|---|---|---|---|
| hostile | 45.8 | 44.1 | 73.1 | – |
| code | 169.0 | 142.8 | 73.0 | 56 (greedy code) |
Acceptance in the hostile phase is ~2 %. Both controllers stay at 15 there. Inside DFlash the best choice would be
off (≈56 tok/s), not 45.
For scale: on this code, a perfect choice would gain about +3 % at 8k (n7 over n15) and nothing at 60k. In drafter-hostile phases it would gain about +24 %. Even with draft depth 0, a loaded DFlash drafter costs ~4 ms per token (56 vs 73 tok/s).
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
Side note: the issue templates still link to the AI policy of ggml-org/llama.cpp, while this repository's own CONTRIBUTING.md allows AI-assisted issues with disclosure.
First Bad Commit
c9e746733 (Complete BeeLlama v0.4.0 upstream rebase) for finding 1. Findings 2 and 3 were not bisected.
Relevant log output
Logs
Controller on main (orig), run B: warm-up request (cold start), first hostile request, and first code request
(new sampling settings, so the stats are reset). These are the only transitions in the whole run. score is the
baseline score in tok/s; the real no-drafter rate is 73 tok/s.
slot apply_adapti: id 0 | task 0 | adaptive draft-max profit: 0 -> 15 (score=38.03)
slot apply_adapti: id 0 | task 32 | adaptive draft-max profit: 0 -> 15 (score=14.98)
slot apply_adapti: id 0 | task 836 | adaptive draft-max profit: 0 -> 15 (score=15.15)
Temporary tracing (LLAMA_DM_TRACE, not part of main) with exploration re-wired:
DM-TRACE baseline cycle_ms=30.21 first_after_prompt=0 baseline_score=36.75 samples=4 n_max=15
DM-TRACE baseline cycle_ms=161.98 first_after_prompt=1 baseline_score=0.00 samples=1 n_max=0
DM-TRACE baseline cycle_ms=29.82 first_after_prompt=0 baseline_score=0.00 samples=2 n_max=0
DM-TRACE spec req=12 n_draft=12 acc=3 draft_ms=5.73 verify_ms=1.55 cycle_ms=45.84 n_max=15
DM-TRACE spec req=8 n_draft=8 acc=2 draft_ms=5.34 verify_ms=1.53 cycle_ms=37.40 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=1 draft_ms=5.33 verify_ms=1.18 cycle_ms=27.62 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=0 draft_ms=5.51 verify_ms=0.79 cycle_ms=26.84 n_max=15
- Lenguaje dominante
- C++
- Estrellas
- 1.2k
- Forks
- 80
- Merge medio
- 4 d 20 h
- PR fusionados (30 d)
- 2
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
Anbeeld/beellama.cpp#205 ·
-
Dificultad 3/5 Medio día Aptitud para principiantes 58/100
Anbeeld/beellama.cpp#206 ·
-
Dificultad 3/5 1-2 días Aptitud para principiantes 57/100
Anbeeld/beellama.cpp#204 ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
Anbeeld/beellama.cpp#203 · 2 comentarios ·
-
Dificultad 4/5 3-5 días Aptitud para principiantes 45/100
Anbeeld/beellama.cpp#202 · 1 comentario ·
Todos los issues de Anbeeld/beellama.cpp
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
CachyOS/packageinstaller#144 ·
Los mantenedores suelen responder en 1 día
-
documentation
Dificultad 1/5 Menos de una hora Aptitud para principiantes 82/100
TomerAberbach/profiler-md#951 ·
Los mantenedores suelen responder en 1 día
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysPosiblemente ocupada @0z5a la tomó hoy. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
apache/pulsar-client-go#1575 ·
Los mantenedores suelen responder en 2 días
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
pocoproject/poco#5764 ·
Los mantenedores suelen responder en 2 días
-
bug cpp-core-guidelines low-priority
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
NatronGitHub/openfx-supportext#164 ·
Los mantenedores suelen responder en 1 día