Misc. bug: DFlash profit draft-max controller never leaves max depth (exploration not wired since v0.4.0 rebase, biased cycle timings)
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 18/100
- issue の種類
- バグ
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 領域
- backend, performance
調査の方向性
The problem sits in tools/server/server-context.cpp (server_slot::get_n_draft_max() near line 719, and the DONE_PROMPT -> GENERATING transition near line 5527) and in server-adaptive-dm.h, where the unused profit_* probe helpers live. The first step is to reproduce the stuck draft-max with --spec-dm-controller profit and the traces described. Done would be a fix for the first-token baseline timing and a controller that can move off depth 15, backed by a test; the CUDA graph key in ggml/src/ggml-cuda/ggml-cuda.cu is a separate, deeper change.
索引モデルが issue の本文から書いたものです。
説明
Name and Version
version: 0.4.9-dev (build 12930, commit b46c53e54)
built with MSVC 19.51.36256.0 for Windows AMD64
CUDA build, CMAKE_CUDA_ARCHITECTURES=120, GGML_CUDA_FA=ON.
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server -m Qwen3.5-4B-Q8_0.gguf -md qwen35-4b-dflash-Q8_0.gguf --spec-type draft-dflash -ngl 99 -c 65536 -b 512 -ub 512 -fa on -np 1 --spec-dm-controller profit --load-mode none -lv 4 --reasoning-loop-guard off
Target unsloth/Qwen3.5-4B-GGUF Q8_0, drafter Anbeeld/Qwen3.5-4B-DFlash-GGUF Q8_0, RTX 5060 Ti 16 GB (sm_120),
PCIe 3.0 x8.
Problem description & steps to reproduce
With --spec-dm-controller profit, DFlash1 holds draft-max 15 for the whole session. It does this even when
speculation is clearly a loss. Three separate causes were found:
1. Depth exploration is no longer scheduled.
server_slot::get_n_draft_max() (tools/server/server-context.cpp:719) only returns adaptive_n_max. The excursions that get_dflash_n_draft_max() used to schedule (4039a8ffd, 57da314b5) were dropped in the v0.4.0 rebase c9e746733, together with tests/test-dflash-plumbing.cpp. The helpers are still in server-adaptive-dm.h, but nothing calls them: profit_next_unready_probe_depth, profit_next_unready_explore_depth, profit_should_probe_baseline, profit_mark_baseline_probe,
profit_next_off_probe_depth.
As a result:
- after the cold-start baseline, no depth other than the held one is ever measured;
- periodic baseline probes never happen;
- the off state has no reprobe.
The controller can still step down through the survival estimate, but only when the held depth is clearly worse than the baseline (hold_episode_clearly_bad).
Claude re-wired the helpers in get_n_draft_max() (baseline probe, off-state reprobe, an excursion every 12 cycles). The controller then does measure other depths, but it still never switches. The next two findings explain why.
2. The first token of every request is recorded as a no-spec baseline cycle.
adaptive_cycle_start_us is only set for slots that are already generating (server-context.cpp:4260). The token sampled right after SLOT_STATE_DONE_PROMPT -> GENERATING (:5527) goes through the no-spec timing path (:5558). Its cycle time is measured from the last cycle of the previous request, so it includes the idle gap and prompt processing. With temporary tracing, these samples are 143–164 ms, while normal baseline cycles are 21–34 ms. profit_score_for_stats() works on totals and there are only a few baseline samples (3–8 here), so one such sample pulls the baseline score from ~38 down to 13–15 tok/s (see log). Likely fix (not built or measured): set adaptive_cycle_start_us = 0 on the DONE_PROMPT -> GENERATING transition.
3. Sporadic cycles at a different depth run without CUDA graphs.
ggml_cuda_graph_get_key() (ggml/src/ggml-cuda/ggml-cuda.cu:2846) returns cgraph->nodes[0], which is the same for every verify batch size. Each shape change resets warmup_complete, so a single excursion or baseline cycle runs eagerly. Mean cycle time per requested depth (traced, 8k):
| cycle | graphs on | GGML_CUDA_DISABLE_GRAPHS=1 |
key incl. nodes[0]->ne (experiment) |
|---|---|---|---|
| depth 15, steady (n≈900) | 28.6 ms | 36.3 | 27.9 |
| depths 1–13 (3–27 samples each) | 35–38 | 36–39 | 31–37 |
| baseline depth 0 (n=12) | 28.0 | 30.0 | 27.3 |
Excursions therefore look ~30 % more expensive than they would be at steady state, and a shallower depth can practically never win. A per-shape graph key alone does not fix this, because a single cycle never gets past warmup and capture. Excursions would probably have to last several cycles, with the first samples discarded.
Measurements. The prompt is the first N tokens of a large Python file (django.py). In the greedy runs each segment continues the original text (prompt = N + i·384 tokens, 5×384 generated), so every variant sees identical inputs. "orig" = main b46c53e54; "explore" = main with the exploration re-wired (finding 1 only).
A. Greedy code, tok/s:
| ctx | orig | explore | static n15 | static n7 | static n3 |
|---|---|---|---|---|---|
| 8k | 171.1 | 162.4 | 170.5 | 175.4 | 146.5 |
| 60k | 318.4 | 310.2 | 317.6 | 245.6 | 146.7 |
Both controllers stay at 15. Exploration costs 2–5 % and never pays off here.
B. Drafter-hostile text (temperature 2.0, top-k 40, 4×256 tokens), then greedy code (4×384), 8k, tok/s:
| phase | orig | explore | no drafter | DFlash loaded, --spec-draft-n-max 0 |
|---|---|---|---|---|
| hostile | 45.8 | 44.1 | 73.1 | – |
| code | 169.0 | 142.8 | 73.0 | 56 (greedy code) |
Acceptance in the hostile phase is ~2 %. Both controllers stay at 15 there. Inside DFlash the best choice would be
off (≈56 tok/s), not 45.
For scale: on this code, a perfect choice would gain about +3 % at 8k (n7 over n15) and nothing at 60k. In drafter-hostile phases it would gain about +24 %. Even with draft depth 0, a loaded DFlash drafter costs ~4 ms per token (56 vs 73 tok/s).
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
Side note: the issue templates still link to the AI policy of ggml-org/llama.cpp, while this repository's own CONTRIBUTING.md allows AI-assisted issues with disclosure.
First Bad Commit
c9e746733 (Complete BeeLlama v0.4.0 upstream rebase) for finding 1. Findings 2 and 3 were not bisected.
Relevant log output
Logs
Controller on main (orig), run B: warm-up request (cold start), first hostile request, and first code request
(new sampling settings, so the stats are reset). These are the only transitions in the whole run. score is the
baseline score in tok/s; the real no-drafter rate is 73 tok/s.
slot apply_adapti: id 0 | task 0 | adaptive draft-max profit: 0 -> 15 (score=38.03)
slot apply_adapti: id 0 | task 32 | adaptive draft-max profit: 0 -> 15 (score=14.98)
slot apply_adapti: id 0 | task 836 | adaptive draft-max profit: 0 -> 15 (score=15.15)
Temporary tracing (LLAMA_DM_TRACE, not part of main) with exploration re-wired:
DM-TRACE baseline cycle_ms=30.21 first_after_prompt=0 baseline_score=36.75 samples=4 n_max=15
DM-TRACE baseline cycle_ms=161.98 first_after_prompt=1 baseline_score=0.00 samples=1 n_max=0
DM-TRACE baseline cycle_ms=29.82 first_after_prompt=0 baseline_score=0.00 samples=2 n_max=0
DM-TRACE spec req=12 n_draft=12 acc=3 draft_ms=5.73 verify_ms=1.55 cycle_ms=45.84 n_max=15
DM-TRACE spec req=8 n_draft=8 acc=2 draft_ms=5.34 verify_ms=1.53 cycle_ms=37.40 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=1 draft_ms=5.33 verify_ms=1.18 cycle_ms=27.62 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=0 draft_ms=5.51 verify_ms=0.79 cycle_ms=26.84 n_max=15
- 主要言語
- C++
- スター
- 1.2k
- フォーク
- 80
- 平均マージ
- 4日 20時間
- マージ済み PR(30日)
- 2
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
Anbeeld/beellama.cpp のほかの issue
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
Anbeeld/beellama.cpp#205 ·
-
難易度 3/5 半日 初心者へのやさしさ 58/100
Anbeeld/beellama.cpp#206 ·
-
難易度 3/5 1〜2日 初心者へのやさしさ 57/100
Anbeeld/beellama.cpp#204 ·
-
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
Anbeeld/beellama.cpp#203 · コメント 2 件 ·
-
難易度 4/5 3〜5日 初心者へのやさしさ 45/100
Anbeeld/beellama.cpp#202 · コメント 1 件 ·
Anbeeld/beellama.cpp の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
CachyOS/packageinstaller#144 ·
メンテナーはふだん 1 日以内に返信
-
documentation
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
TomerAberbach/profiler-md#951 ·
メンテナーはふだん 1 日以内に返信
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keys対応中かも @0z5a が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
apache/pulsar-client-go#1575 ·
メンテナーはふだん 2 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
pocoproject/poco#5764 ·
メンテナーはふだん 2 日以内に返信
-
bug cpp-core-guidelines low-priority
難易度 2/5 1〜3時間 初心者へのやさしさ 62/100
NatronGitHub/openfx-supportext#164 ·
メンテナーはふだん 1 日以内に返信