[Bug] DFlash 2 speculative decoding crashes and fails in Python due to missing ctx_other and non-autoregressive graph execution
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Domain
- ai, ai-infra-agents, backend
Research direction
The issue is in llama_cpp/llama.py and _internals.py where DFlash 2 draft model initialization fails due to missing ctx_other linkage. Examine the C++ fork at z-lab/llama.cpp-fork (commit 5ecbe1ac) to understand the native block-diffusion pipeline. The fix likely involves exposing ctx_other in the Python bindings and ensuring the draft model subclass integrates with the C++ speculative decoding flow. Test with the provided Qwen3.8-27B models to verify draft acceptance rates improve.
Written by the indexing model from the issue text.
Description
Prerequisites
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
When loading the target model utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF (Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf) together with the DFlash 2 draft model incoai/Qwen3.8-27B-DFlash2-GGUF (Qwen3.8-27B-DFlash2-Q4_K_M.gguf), llama-cpp-python should link the draft context to the target context via cparams.ctx_other = target_context and execute the native C++ block-diffusion speculative decoding pipeline, producing speculative speedups.
Current Behavior
- Calling
draft_llm = Llama(model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf")fails in GGML with:
dflash requires ctx_other to be set->ValueError: Failed to create llama_context
because DFlash 2 sidecar GGUF files do not contain their ownlm_head/output.weighttensors and requirectx_otherto be set during context initialization. - Setting
draft_llm.model = draft_model_ptrfails with:
AttributeError: property 'model' of 'Llama' object has no setter. - Calling
LlamaDraftModel(draft_llm, num_pred_tokens=5)fails with:
TypeError: LlamaDraftModel() takes no argumentsbecauseLlamaDraftModelis an abstract base class. - When subclassing
LlamaDraftModeland returning a Pythonlist, it crashes insidellama.pywith:
AttributeError: 'list' object has no attribute 'astype'. - When returning a
numpy.ndarraywithdtype=np.intc, the Python loop callsdraft_llm.sample(). This bypasses the C++ DFlash 2 pipeline (target hidden layer extractionllama_get_embeddings_layer_inp, encoder passllama_encode, and candidate lattice selectionbuild_post_sampling), resulting in a 0% draft acceptance rate and dropping generation speed to baseline autoregressive throughput (~34.27 tok/s).
Environment and Context
- Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition (
sm_120), 24+ GB VRAM - Environment: Hugging Face ZeroGPU Space
- Operating System: Debian GNU/Linux 13 (trixie) / Linux 6.12.94-123.192.amzn2023.x86_64
- glibc Version: Debian GLIBC 2.41-12
- Python Version: 3.12.12
- NVIDIA Driver: 580.159.03 (CUDA 13.0)
- llama-cpp-python: v0.3.35 compiled against submodule
vendor/llama.cppusing:- Repository: https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4
- Branch:
dflash2 - Commit Hash:
5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 - Upstream Pull Request: https://github.com/ggml-org/llama.cpp/pull/27342/changes
Steps to Reproduce
- Build
llama-cpp-pythonwithvendor/llama.cppchecked out atz-lab/llama.cpp-forkcommit5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4. - Download target model
utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUFand draft modelincoai/Qwen3.8-27B-DFlash2-GGUF. - Attempt to initialize the draft model in Python:
import llama_cpp
from llama_cpp import Llama
# 1. Target model initialization succeeds:
target_llm = Llama(
model_path="Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)
# 2. Draft model initialization fails here:
draft_llm = Llama(
model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf",
n_gpu_layers=-1,
n_ctx=8192,
)
Failure Logs
Context creation failure:
File "app.py", line 58, in get_or_load_model
draft_llm = Llama(model_path=DRAFT_MODEL_PATH, ...)
File ".../llama_cpp/llama.py", line 415, in __init__
internals.LlamaContext(
File ".../llama_cpp/_internals.py", line 266, in __init__
raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context
Attribute setter failure:
File "app.py", line 94, in get_or_load_dflash2_model
draft_llm.model = draft_model_ptr
AttributeError: property 'model' of 'Llama' object has no setter
LlamaDraftModel constructor failure:
File "app.py", line 108, in get_or_load_dflash2_model
target_llm.draft_model = LlamaDraftModel(draft_model=draft_llm, num_pred_tokens=5)
TypeError: LlamaDraftModel() takes no arguments
List vs NumPy array failure:
File ".../llama_cpp/llama.py", line 1022, in generate
draft_tokens.astype(int)[
AttributeError: 'list' object has no attribute 'astype'
Additional Notes & Disclaimers
- Goal: I am trying to run DFlash 2 (
Qwen3.8-27B-DFlash2-Q4_K_M.gguf) on a Hugging Face ZeroGPU Space withllama-cpp-python. - Specific Fork: The C++ code is from https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (PR https://github.com/ggml-org/llama.cpp/pull/27342/changes).
- Disclaimer on other speculative methods: The mention of EAGLE was suggested by an AI assistant during our discussion; I cannot personally confirm EAGLE's implementation details. I also do not know with certainty about built-in MTP, DFlash v1, or DSpark in
llama-cpp-python, though DFlash v1 and DSpark may already be present in upstreamllama.cpp. - AI Disclosure: This bug report was formatted with the help of an AI assistant at my request during a live debugging session. All steps, code snippets, stack traces, and reproduction logs come directly from my testing on Hugging Face ZeroGPU.
- Dominant language
- Python
- Stars
- 10.7k
- Forks
- 1.5k
- Avg merge
- 1h 40m
- Merged PRs (30d)
- 5
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from abetlen/llama-cpp-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
abetlen/llama-cpp-python#2424 ·
Maintainers usually reply within 1 day
-
Seven llama_sampler_init_* bindings admit keyword arguments that the ctypes function object silently dropsPossibly taken @Belal0066 claimed this 59 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
abetlen/llama-cpp-python#2371 ·
Maintainers usually reply within 1 day
-
uv add llama-cpp-python wheels fails for versions above 0.3.30Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2352 · 1 comment · 3 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
abetlen/llama-cpp-python#2314 ·
Maintainers usually reply within 1 day
-
Llama.embed() calls LlamaBatch.add_sequence with old 3-arg signature; missing logits_arrayMay be free again @lxcxjxhx claimed this 136 days ago, and no pull request is open. Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
abetlen/llama-cpp-python#2211 · 2 comments ·
Maintainers usually reply within 1 day
All issues in abetlen/llama-cpp-python
Similar issues
-
Bug IronPython Python 3
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pyrevitlabs/pyRevit#4184 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 74/100
jqueryscript/awesome-claude-code#1009 ·
-
tests
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
[Plugin] 群聊记忆 · 人格养成Open
Difficulty 1/5 1-3 hours Newbie friendliness 74/100