Misc. bug: DFlash profit draft-max controller never leaves max depth (exploration not wired since v0.4.0 rebase, biased cycle timings)
Chưa có ai nhận issue này.
Đánh giá
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức phù hợp với người mới
- 18/100
- Loại issue
- Lỗi
- Độ rõ ràng
- Khá rõ ràng
- Mức độ hoạt động
- Sôi nổi
- Lĩnh vực
- backend, performance
Hướng nghiên cứu
The problem sits in tools/server/server-context.cpp (server_slot::get_n_draft_max() near line 719, and the DONE_PROMPT -> GENERATING transition near line 5527) and in server-adaptive-dm.h, where the unused profit_* probe helpers live. The first step is to reproduce the stuck draft-max with --spec-dm-controller profit and the traces described. Done would be a fix for the first-token baseline timing and a controller that can move off depth 15, backed by a test; the CUDA graph key in ggml/src/ggml-cuda/ggml-cuda.cu is a separate, deeper change.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Mô tả
Name and Version
version: 0.4.9-dev (build 12930, commit b46c53e54)
built with MSVC 19.51.36256.0 for Windows AMD64
CUDA build, CMAKE_CUDA_ARCHITECTURES=120, GGML_CUDA_FA=ON.
Operating systems
Windows
Which llama.cpp modules do you know to be affected?
llama-server
Command line
llama-server -m Qwen3.5-4B-Q8_0.gguf -md qwen35-4b-dflash-Q8_0.gguf --spec-type draft-dflash -ngl 99 -c 65536 -b 512 -ub 512 -fa on -np 1 --spec-dm-controller profit --load-mode none -lv 4 --reasoning-loop-guard off
Target unsloth/Qwen3.5-4B-GGUF Q8_0, drafter Anbeeld/Qwen3.5-4B-DFlash-GGUF Q8_0, RTX 5060 Ti 16 GB (sm_120),
PCIe 3.0 x8.
Problem description & steps to reproduce
With --spec-dm-controller profit, DFlash1 holds draft-max 15 for the whole session. It does this even when
speculation is clearly a loss. Three separate causes were found:
1. Depth exploration is no longer scheduled.
server_slot::get_n_draft_max() (tools/server/server-context.cpp:719) only returns adaptive_n_max. The excursions that get_dflash_n_draft_max() used to schedule (4039a8ffd, 57da314b5) were dropped in the v0.4.0 rebase c9e746733, together with tests/test-dflash-plumbing.cpp. The helpers are still in server-adaptive-dm.h, but nothing calls them: profit_next_unready_probe_depth, profit_next_unready_explore_depth, profit_should_probe_baseline, profit_mark_baseline_probe,
profit_next_off_probe_depth.
As a result:
- after the cold-start baseline, no depth other than the held one is ever measured;
- periodic baseline probes never happen;
- the off state has no reprobe.
The controller can still step down through the survival estimate, but only when the held depth is clearly worse than the baseline (hold_episode_clearly_bad).
Claude re-wired the helpers in get_n_draft_max() (baseline probe, off-state reprobe, an excursion every 12 cycles). The controller then does measure other depths, but it still never switches. The next two findings explain why.
2. The first token of every request is recorded as a no-spec baseline cycle.
adaptive_cycle_start_us is only set for slots that are already generating (server-context.cpp:4260). The token sampled right after SLOT_STATE_DONE_PROMPT -> GENERATING (:5527) goes through the no-spec timing path (:5558). Its cycle time is measured from the last cycle of the previous request, so it includes the idle gap and prompt processing. With temporary tracing, these samples are 143–164 ms, while normal baseline cycles are 21–34 ms. profit_score_for_stats() works on totals and there are only a few baseline samples (3–8 here), so one such sample pulls the baseline score from ~38 down to 13–15 tok/s (see log). Likely fix (not built or measured): set adaptive_cycle_start_us = 0 on the DONE_PROMPT -> GENERATING transition.
3. Sporadic cycles at a different depth run without CUDA graphs.
ggml_cuda_graph_get_key() (ggml/src/ggml-cuda/ggml-cuda.cu:2846) returns cgraph->nodes[0], which is the same for every verify batch size. Each shape change resets warmup_complete, so a single excursion or baseline cycle runs eagerly. Mean cycle time per requested depth (traced, 8k):
| cycle | graphs on | GGML_CUDA_DISABLE_GRAPHS=1 |
key incl. nodes[0]->ne (experiment) |
|---|---|---|---|
| depth 15, steady (n≈900) | 28.6 ms | 36.3 | 27.9 |
| depths 1–13 (3–27 samples each) | 35–38 | 36–39 | 31–37 |
| baseline depth 0 (n=12) | 28.0 | 30.0 | 27.3 |
Excursions therefore look ~30 % more expensive than they would be at steady state, and a shallower depth can practically never win. A per-shape graph key alone does not fix this, because a single cycle never gets past warmup and capture. Excursions would probably have to last several cycles, with the first samples discarded.
Measurements. The prompt is the first N tokens of a large Python file (django.py). In the greedy runs each segment continues the original text (prompt = N + i·384 tokens, 5×384 generated), so every variant sees identical inputs. "orig" = main b46c53e54; "explore" = main with the exploration re-wired (finding 1 only).
A. Greedy code, tok/s:
| ctx | orig | explore | static n15 | static n7 | static n3 |
|---|---|---|---|---|---|
| 8k | 171.1 | 162.4 | 170.5 | 175.4 | 146.5 |
| 60k | 318.4 | 310.2 | 317.6 | 245.6 | 146.7 |
Both controllers stay at 15. Exploration costs 2–5 % and never pays off here.
B. Drafter-hostile text (temperature 2.0, top-k 40, 4×256 tokens), then greedy code (4×384), 8k, tok/s:
| phase | orig | explore | no drafter | DFlash loaded, --spec-draft-n-max 0 |
|---|---|---|---|---|
| hostile | 45.8 | 44.1 | 73.1 | – |
| code | 169.0 | 142.8 | 73.0 | 56 (greedy code) |
Acceptance in the hostile phase is ~2 %. Both controllers stay at 15 there. Inside DFlash the best choice would be
off (≈56 tok/s), not 45.
For scale: on this code, a perfect choice would gain about +3 % at 8k (n7 over n15) and nothing at 60k. In drafter-hostile phases it would gain about +24 %. Even with draft depth 0, a loaded DFlash drafter costs ~4 ms per token (56 vs 73 tok/s).
AI Disclosure: The analysis, measurements and this text were prepared with AI assistance (Claude Opus 5.5), reviewed and edited by me.
Side note: the issue templates still link to the AI policy of ggml-org/llama.cpp, while this repository's own CONTRIBUTING.md allows AI-assisted issues with disclosure.
First Bad Commit
c9e746733 (Complete BeeLlama v0.4.0 upstream rebase) for finding 1. Findings 2 and 3 were not bisected.
Relevant log output
Logs
Controller on main (orig), run B: warm-up request (cold start), first hostile request, and first code request
(new sampling settings, so the stats are reset). These are the only transitions in the whole run. score is the
baseline score in tok/s; the real no-drafter rate is 73 tok/s.
slot apply_adapti: id 0 | task 0 | adaptive draft-max profit: 0 -> 15 (score=38.03)
slot apply_adapti: id 0 | task 32 | adaptive draft-max profit: 0 -> 15 (score=14.98)
slot apply_adapti: id 0 | task 836 | adaptive draft-max profit: 0 -> 15 (score=15.15)
Temporary tracing (LLAMA_DM_TRACE, not part of main) with exploration re-wired:
DM-TRACE baseline cycle_ms=30.21 first_after_prompt=0 baseline_score=36.75 samples=4 n_max=15
DM-TRACE baseline cycle_ms=161.98 first_after_prompt=1 baseline_score=0.00 samples=1 n_max=0
DM-TRACE baseline cycle_ms=29.82 first_after_prompt=0 baseline_score=0.00 samples=2 n_max=0
DM-TRACE spec req=12 n_draft=12 acc=3 draft_ms=5.73 verify_ms=1.55 cycle_ms=45.84 n_max=15
DM-TRACE spec req=8 n_draft=8 acc=2 draft_ms=5.34 verify_ms=1.53 cycle_ms=37.40 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=1 draft_ms=5.33 verify_ms=1.18 cycle_ms=27.62 n_max=15
DM-TRACE spec req=15 n_draft=15 acc=0 draft_ms=5.51 verify_ms=0.79 cycle_ms=26.84 n_max=15
- Ngôn ngữ chính
- C++
- Star
- 1.2k
- Fork
- 80
- Merge trung bình
- 4 ngày 20 giờ
- Pull request đã merge (30 ngày)
- 2
Chuẩn bị môi trường
Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.
Bắt đầu từ đâu
- Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
- Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
- Fork repository và làm thay đổi trên một nhánh.
- Mở pull request có tham chiếu số hiệu của issue.
Issue khác của Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotĐang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 68/100
Anbeeld/beellama.cpp#205 ·
-
Độ khó 3/5 Nửa ngày Mức phù hợp với người mới 58/100
Anbeeld/beellama.cpp#206 ·
-
Độ khó 3/5 1-2 ngày Mức phù hợp với người mới 57/100
Anbeeld/beellama.cpp#204 ·
-
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 45/100
Anbeeld/beellama.cpp#203 · 2 bình luận ·
-
Độ khó 4/5 3-5 ngày Mức phù hợp với người mới 45/100
Anbeeld/beellama.cpp#202 · 1 bình luận ·
Tất cả issue của Anbeeld/beellama.cpp
Issue tương tự
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
CachyOS/packageinstaller#144 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
documentation
Độ khó 1/5 Dưới một giờ Mức phù hợp với người mới 82/100
TomerAberbach/profiler-md#951 ·
Maintainer thường phản hồi trong vòng 1 ngày
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysCó thể đã có người làm @0z5a đã nhận hôm nay. Đang mở
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 82/100
apache/pulsar-client-go#1575 ·
Maintainer thường phản hồi trong vòng 2 ngày
-
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 72/100
pocoproject/poco#5764 ·
Maintainer thường phản hồi trong vòng 2 ngày
-
bug cpp-core-guidelines low-priority
Độ khó 2/5 1-3 giờ Mức phù hợp với người mới 62/100
NatronGitHub/openfx-supportext#164 ·
Maintainer thường phản hồi trong vòng 1 ngày