Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug]: reasoning_loop_recovery: recover never fires for a low or medium request (focused_recovery_prompt only matches the template's xhigh sentence)

Open Beginner friendly
  • Is something wrong?
#2,906 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai, backend

Research direction

Start at focused_recovery_prompt() and its caller around serve/server.py:196; the reproduction script in the issue shows how to exercise both prompt positions without an engine. Update recovery to replace the request’s actual effort sentence, with a clear skip message when none matches. Add or run focused tests covering low, medium, and xhigh recovery so the recover branch no longer silently returns without action.

Written by the indexing model from the issue text.

Description

What happened

With "reasoning_loop_recovery": "recover" in a model config, a reply whose thinking repeats whole passages is detected and then does nothing: the server logs the detection and the thinking carries on to reasoning_budget_tokens (or to max_tokens when no budget is set). The reply is not continued with the low-effort instruction, and nothing says why.

Measured on 0.1.44.1 and again on 0.1.45, a community qwen4exp fine-tune (RVN abliterated IQ3_S), reasoning_effort: low, one prompt that makes the model re-verify the same facts, max_tokens 8192, reasoning_budget_tokens 8192 then 4096:

run budget detection what happened next
1 8192 repeated reasoning detected at 3072 tokens (coverage=0.277) thought on to 6144, then the budget wrapped it up
2 8192 repeated reasoning detected at 3072 tokens (coverage=0.289) the same
3 4096 repeated reasoning detected at 3072 tokens (coverage=0.343) thought on to 4096, then the budget wrapped it up

So the detector works (it is the log line above); the recovery step is what never runs.

Why

focused_recovery_prompt() (serve/server.py:196 in 0.1.45) builds the spliced prompt by replacing the template's own xhigh sentence (HIGH_EFFORT, line 160) in the first system message with the low one, and returns None when that sentence is not there as whole tokens:

old, new = tok.encode(HIGH_EFFORT, parse_special=False), tok.encode(LOW_EFFORT, parse_special=False)
for i in range(len(head) - len(old) + 1):
    if head[i:i + len(old)] == old:
        return ids[:i] + new + ids[i + len(old):] + list(generated)
return None

With "effort_position": "start" (the default) a request that asks for low or medium is rendered with that level's sentence at the top of the first system message, so the xhigh sentence is absent and the function returns None. The caller only acts on a non-None result:

recover_prompt = focused_recovery_prompt(self.tok, ids, raw_ids)
if recover_prompt is not None:
    break

recover is therefore a silent no-op for every low/medium request — which is the common case as soon as a config sets "reasoning_effort": "low" (as mine does, to keep a local model from burning its output budget on thinking) or a client picks Low in its own selector. "reasoning_loop_recovery": "stop" is unaffected (it does not use this function and did end the reply in my runs).

How I checked it (no engine needed)

Rendering one request both ways with the server's own functions and then calling focused_recovery_prompt on it:

--- effort_position="start" (default)
    prompt tokens                : 63
    xhigh sentence in the prompt : False
    low   sentence in the prompt : True
    focused_recovery_prompt()    : None -> the recover branch cannot fire
--- effort_position="end"
    prompt tokens                : 105
    xhigh sentence in the prompt : True
    low   sentence in the prompt : True
    focused_recovery_prompt()    : 93 ids -> it can fire

Script: build the tokenizer and the ChatTemplate from a pack's tokenizer/ exactly as main() does, then

low_text  = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=True)          # start
hist      = tpl.render(msgs, add_generation_prompt=False)                                 # end, default (xhigh)
full      = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=True)
mine      = tpl.render(msgs, reasoning_effort="low", add_generation_prompt=False)
end_text  = hist + EFFORT_TURN.format(EFFORT_TEXT["low"]) + full[len(mine):]
for t in (low_text, end_text):
    print(focused_recovery_prompt(tok, tok.encode(t, parse_special=True), []))
Workaround I use now

"effort_position": "end" in the model config: the history is then rendered with the template's default (xhigh) and the real level follows in a short system turn before the answer, so the xhigh sentence is in the prompt and the recovery can splice it. With it, the detection line is followed by resuming the same output with the low-effort instruction (reasoning_loop_recovery) (measured twice, coverage 0.277 and 0.343).

Two honest notes on that: the nudge is not a cure — in both runs the model went on thinking for another ~1000-3000 tokens to the budget — and effort_position: "end" is a separate opt-in with its own behaviour (the engine checkpoints in front of the tail turn).

Suggested fix

Pass the request's level into focused_recovery_prompt() and look for that sentence (the template knows low, medium, xhigh; high maps to xhigh), or fall back to the low sentence when the level's own is not found. If neither is found, print one line saying the recovery was skipped and why, instead of failing silently — the current behaviour costs a full thinking budget with no hint.

Environment

0.1.44.1 and 0.1.45, prebuilt CUDA 13.0 engine (sm_75/86/89/120) · RTX 5080 16 GB · Windows 11 26200 · Core Ultra 7 270K Plus, 95 GB RAM. Config: reasoning_effort: low, reasoning_budget_tokens: 4096/8192, reasoning_loop_recovery: recover, reasoning_close_retry: true.

Not tested

AMD HIP and SYCL (CUDA only here); a high/xhigh request under effort_position: "start" (that one renders the xhigh sentence at the top, so the recovery should fire — I did not run it); the Anthropic route's output_config.effort path; and whether the model's repetition after the nudge would end on its own with a different prompt.

Dominant language
C++
Stars
11.6k
Forks
1k
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Niko1221/Strata

All issues in Niko1221/Strata

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.