Eval bug: reasoning loop guard force-closes legitimate reasoning early (looks like reasoning budget exhausted)
Ninguém assumiu esta issue ainda.
Avaliação
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Facilidade para iniciantes
- 45/100
- Tipo de issue
- Bug
- Clareza
- Razoavelmente clara
- Status de atividade
- Ativa
- Stack de tecnologia
- cpp
- Domínio
- ai-infra-agents
Direção de pesquisa
Start at the /v1/chat/completions entry point and trace how reasoning_loop_guard, reasoning_loop_min_tokens, and the force-close message interact with reasoning-token accounting. Reproduce the reported early cut with the provided 8000-token budget and compare against guard-off behavior. Done means legitimate restatements do not trigger premature closure, actual budget limits still apply, and reasoning-token usage remains accurate when the guard is off.
Escrita pelo modelo de indexação a partir do texto da issue.
Descrição
Name and Version
llama-server version: 0.4.9 (build 12930, commit b46c53e54), Windows x64 CUDA 13.3 release binary, router mode.
Operating systems
Windows 11
GGML backends
CUDA
Hardware
RTX 4090 24 GB
Models
unsloth Qwen3.8-27B-GGUF, UD-Q4_K_XL (both the August upload and Dynamic v3.1). The same thing happens with UD-IQ4_XS. Settings: --spec-type draft-mtp --spec-draft-n-max 3, KV kvarn6, --reasoning-budget 8000, --reasoning-budget-message "... reasoning budget exceeded, need to answer.\n", reasoning_effort: medium.
Problem description & steps to reproduce
With the default --reasoning-loop-guard force-close, hidden reasoning is often force-closed well before the reasoning budget is reached. With a budget of 8000, we saw the budget message injected after only about 1.1K–1.9K reasoning tokens. Because the force-close uses the budget message, it looks like a broken budget counter. Budgets of 500 and 2000 are honoured exactly.
All the early cuts we looked at happen at the same point: the model re-states content it wrote earlier in the same reasoning, on purpose. In our case it re-copies a 3×3 OKLab matrix with 10-digit coefficients to double-check it. That is legitimate verification, not a degenerate loop. It is also a very predictable stretch, where MTP drafts get accepted almost entirely.
Test setup: one coding prompt (implement OKLab colour mixing, stdlib only, with stated edge cases), thinking_budget_tokens: 8000, non-streaming /v1/chat/completions, temp 1.0 / top_p 0.95 / top_k 20.
| per-request setting | early cut (budget message in reasoning) | usage.completion_tokens_details.reasoning_tokens |
|---|---|---|
| default | 3/6 (at 1100–1930 tokens) | reported |
"reasoning_loop_guard": "off" |
0/6 | always 0 |
"reasoning_loop_window": 2048, "reasoning_loop_min_coverage": 1024, "reasoning_loop_min_tokens": 2048 |
7/8, six of them at exactly 2060 tokens (the first check after min_tokens) |
reported |
"reasoning_loop_min_tokens": 100000 |
0/6 (2.8K–5.4K tokens of reasoning, finishing normally) | reported |
An earlier run with the same prompt (budget 8000, 6 requests per model version) cut 4/6 on the August GGUF and 1/6 on v3.1. In a small coding benchmark, the share of tasks with all tests passing dropped from 97 % to 74 %, and every failure traced back to one of these early cuts.
Observations / possible issues:
- False positives on legitimate restatements. A single re-copy of a ~200–300-token block is enough to trigger. A larger window and a higher coverage threshold make it worse (7/8). That suggests the detector counts one repetition of a long span as "coverage", not a repetition that happens many times.
- Misleading message. The force-close injects the reasoning-budget message. From the client side, a loop-guard intervention can't be told apart from an exhausted budget. The
loop_guardtelemetry block did not appear in the OpenAI-compatible chat completion response, so we only found the cause by reading the source. reasoning_loop_guard: offalso turns off reasoning-token accounting (reasoning_tokens= 0 in usage), apparently becausereasoning_output_tokensis only updated in the loop-guard accept callback.
Suggestions:
- Require several repetitions (period × count) before triggering, instead of a single restated span.
- Use a separate message, or none, for loop-guard force-close, and expose the
loop_guardevent in chat-completion responses. - Keep reasoning-token accounting independent of the guard mode.
Workaround in use: reasoning-loop-min-tokens = 100000 in the preset. The reasoning budget still caps real loops.
First Bad Commit
Not bisected. The behaviour is present in v0.4.9.
Relevant log output
Example tail of reasoning_content at a cut (budget 8000, 1484 reasoning tokens):
r = 4.0767416621f * l - 3.3077115913f * m + 0.2309699292f * s
g = -... reasoning budget exceeded, need to answer.\n
- Linguagem predominante
- C++
- Estrelas
- 1.2k
- Forks
- 80
- Merge médio
- 4d 20h
- PRs com merge (30d)
- 2
Preparar o ambiente
Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Mais de Anbeeld/beellama.cpp
-
[BUG] KVarN assertion failure in build_inp_mem_hybrid_iswa() due to uninitialized self_kvarn_rotAberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 68/100
Anbeeld/beellama.cpp#205 ·
-
Dificuldade 3/5 Meio dia Facilidade para iniciantes 58/100
Anbeeld/beellama.cpp#206 ·
-
Dificuldade 3/5 1-2 dias Facilidade para iniciantes 57/100
Anbeeld/beellama.cpp#204 ·
-
Dificuldade 4/5 3-5 dias Facilidade para iniciantes 45/100
Anbeeld/beellama.cpp#203 · 2 comentários ·
-
Dificuldade 4/5 3-5 dias Facilidade para iniciantes 48/100
Anbeeld/beellama.cpp#200 ·
Todas as issues de Anbeeld/beellama.cpp
Issues semelhantes
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
CachyOS/packageinstaller#144 ·
Mantenedores costumam responder em até 1 dia
-
documentation
Dificuldade 1/5 Menos de uma hora Facilidade para iniciantes 82/100
TomerAberbach/profiler-md#951 ·
Mantenedores costumam responder em até 1 dia
-
[Bug] Default key routing (JavaStringHash) picks a different partition than the Java client for about half of keysTalvez já em andamento @0z5a assumiu hoje. Aberta
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 82/100
apache/pulsar-client-go#1575 ·
Mantenedores costumam responder em até 2 dias
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
pocoproject/poco#5764 ·
Mantenedores costumam responder em até 2 dias
-
bug cpp-core-guidelines low-priority
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 62/100
NatronGitHub/openfx-supportext#164 ·
Mantenedores costumam responder em até 1 dia