ds4-eval: strict grader marks correct answer INCOMPLETE when output ends with "</think>E" (no "Answer:" line)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- c
- Domain
- testing-qa
Research direction
Start in ds4_eval.c at find_case_answer_strict() around line 3323, and compare its last-line handling with the lenient find_answer_letter() extractor. Run the deterministic reproduction command in the issue before and after the change. Done means the glued </think>E case is graded as E rather than INCOMPLETE, without changing the behavior for other outputs.
Written by the indexing model from the issue text.
Description
Summary
find_case_answer_strict() requires the final line of the output to contain an
Answer: marker. When the model closes thinking and emits the bare answer glued
to the tag (e.g. ...</think>E), the grader returns INCOMPLETE even though the
letter is correct.
Repro (deterministic)
./ds4-eval --plain --case-id 4a1d1780a93f4093b6fb7d3c314cbea8
Model: DeepSeek V4 Flash Q2 (ds4flash.gguf), Metal, M5 Max 128GB, temp 0
(eval default). The model deterministically ends its output with:
...So answer E.</think>E
Result: INCOMPLETE got ? expected E. --retry-incomplete does not help — the
retry produces the same 183-token output.
Root cause
find_case_answer_strict() (ds4_eval.c, ~line 3323):
- trims trailing whitespace, takes the last line;
- requires
answer_payload(line)(an "Answer:" marker) — otherwise returns
false →EVAL_INCOMPLETE.
The lenient extractor (find_answer_letter() reverse scan) would catch the
bare E, but it is only used by --regrade-trace, not live runs.
Suggested fix
In the strict path, when the last line has no Answer: marker but contains
</think> followed by non-empty text, grade the text after the tag. A minimal
local patch of ~10 lines does this and flips the case to PASSED.
Why it matters
INCOMPLETE is meant to separate "did not follow the format" from "answered
wrong". A correct bare answer glued to </think> is a common, benign deviation;
counting it as INCOMPLETE understates the model and wastes a retry budget.
- Dominant language
- C
- Stars
- 23.6k
- Forks
- 2.3k
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from antirez/ds4
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
antirez/ds4#1189 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 92/100
antirez/ds4#1032 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
antirez/ds4#1019 · 1 comment · 1 reaction ·
Maintainers usually reply within 1 day
-
ds4-server: no reply-budget or forced think-close, so HTTP callers can hang where ds4-eval terminatesPossibly taken @paride-cmd claimed this 35 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
Similar issues
-
rc_runtime_activate_richpresence leaves a half-initialised entry when the buffer allocation failsOpen
Difficulty 1/5 Under an hour Newbie friendliness 88/100
RetroAchievements/rcheevos#558 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
libsdl-org/SDL#16464 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
chore(gateway): emit INFO budget reserved/settled logs for proactivity v2 (chip task_2855f4ec)Possibly taken A pull request linked to this issue is open or already merged. Openbackend
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
BasedHardware/omi#20940 ·
Maintainers usually reply within 1 day