Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Bug] DFlash 2 speculative decoding crashes and fails in Python due to missing ctx_other and non-autoregressive graph execution

Aberta
  • Algo errado?
#2,361 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
25/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Pouca atividade
Stack de tecnologia
c, python
Domínio
ai, ai-infra-agents, backend

Direção de pesquisa

A issue está em llama_cpp/llama.py e _internals.py, onde a inicialização do modelo draft DFlash 2 falha devido à ausência da vinculação com ctx_other. Examine o fork de C++ em z-lab/llama.cpp-fork (commit 5ecbe1ac) para entender o pipeline nativo de block-diffusion. A correção provavelmente envolve expor ctx_other nos bindings de Python e garantir que a subclasse do modelo draft se integre ao fluxo de speculative decoding em C++. Teste com os modelos Qwen3.8-27B fornecidos para verificar se as taxas de aceitação do draft melhoram.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Prerequisites

  • I am running the latest code. Development is very rapid so there are no tagged versions as of now.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

When loading the target model utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF (Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf) together with the DFlash 2 draft model incoai/Qwen3.8-27B-DFlash2-GGUF (Qwen3.8-27B-DFlash2-Q4_K_M.gguf), llama-cpp-python should link the draft context to the target context via cparams.ctx_other = target_context and execute the native C++ block-diffusion speculative decoding pipeline, producing speculative speedups.

Current Behavior

  1. Calling draft_llm = Llama(model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf") fails in GGML with:
    dflash requires ctx_other to be set -> ValueError: Failed to create llama_context
    because DFlash 2 sidecar GGUF files do not contain their own lm_head / output.weight tensors and require ctx_other to be set during context initialization.
  2. Setting draft_llm.model = draft_model_ptr fails with:
    AttributeError: property 'model' of 'Llama' object has no setter.
  3. Calling LlamaDraftModel(draft_llm, num_pred_tokens=5) fails with:
    TypeError: LlamaDraftModel() takes no arguments because LlamaDraftModel is an abstract base class.
  4. When subclassing LlamaDraftModel and returning a Python list, it crashes inside llama.py with:
    AttributeError: 'list' object has no attribute 'astype'.
  5. When returning a numpy.ndarray with dtype=np.intc, the Python loop calls draft_llm.sample(). This bypasses the C++ DFlash 2 pipeline (target hidden layer extraction llama_get_embeddings_layer_inp, encoder pass llama_encode, and candidate lattice selection build_post_sampling), resulting in a 0% draft acceptance rate and dropping generation speed to baseline autoregressive throughput (~34.27 tok/s).

Environment and Context

Steps to Reproduce

  1. Build llama-cpp-python with vendor/llama.cpp checked out at z-lab/llama.cpp-fork commit 5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4.
  2. Download target model utautako/Qwen3.8-27B-NVFP4-MTP-Q8attn-GGUF and draft model incoai/Qwen3.8-27B-DFlash2-GGUF.
  3. Attempt to initialize the draft model in Python:
import llama_cpp
from llama_cpp import Llama

# 1. Target model initialization succeeds:
target_llm = Llama(
    model_path="Qwen3.8-27B-NVFP4-MTP-Q8attn.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
)

# 2. Draft model initialization fails here:
draft_llm = Llama(
    model_path="Qwen3.8-27B-DFlash2-Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
)

Failure Logs

Context creation failure:

  File "app.py", line 58, in get_or_load_model
    draft_llm = Llama(model_path=DRAFT_MODEL_PATH, ...)
  File ".../llama_cpp/llama.py", line 415, in __init__
    internals.LlamaContext(
  File ".../llama_cpp/_internals.py", line 266, in __init__
    raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context

Attribute setter failure:

  File "app.py", line 94, in get_or_load_dflash2_model
    draft_llm.model = draft_model_ptr
AttributeError: property 'model' of 'Llama' object has no setter

LlamaDraftModel constructor failure:

  File "app.py", line 108, in get_or_load_dflash2_model
    target_llm.draft_model = LlamaDraftModel(draft_model=draft_llm, num_pred_tokens=5)
TypeError: LlamaDraftModel() takes no arguments

List vs NumPy array failure:

  File ".../llama_cpp/llama.py", line 1022, in generate
    draft_tokens.astype(int)[
AttributeError: 'list' object has no attribute 'astype'

Additional Notes & Disclaimers

  • Goal: I am trying to run DFlash 2 (Qwen3.8-27B-DFlash2-Q4_K_M.gguf) on a Hugging Face ZeroGPU Space with llama-cpp-python.
  • Specific Fork: The C++ code is from https://github.com/z-lab/llama.cpp-fork/tree/5ecbe1ac17ec0484c5b44af0bd580cdc9c428ed4 (PR https://github.com/ggml-org/llama.cpp/pull/27342/changes).
  • Disclaimer on other speculative methods: The mention of EAGLE was suggested by an AI assistant during our discussion; I cannot personally confirm EAGLE's implementation details. I also do not know with certainty about built-in MTP, DFlash v1, or DSpark in llama-cpp-python, though DFlash v1 and DSpark may already be present in upstream llama.cpp.
  • AI Disclosure: This bug report was formatted with the help of an AI assistant at my request during a live debugging session. All steps, code snippets, stack traces, and reproduction logs come directly from my testing on Hugging Face ZeroGPU.
Linguagem predominante
Python
Estrelas
10.6k
Forks
1.5k
Merge médio
3h 57min
PRs com merge (30d)
4

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de abetlen/llama-cpp-python

Todas as issues de abetlen/llama-cpp-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.