Eval judge: ask for the reason before the verdict; 4 verdicts in 2 receipts contradict their own reasons

Open
  • Is something wrong?
#306 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start by reading JUDGE_PROMPT in evals/ab_harness.py and the report sections and regression test cited in the issue. Change the JSON field order so reason comes before the verdict fields, then re-judge a recent receipt with both prompts and compare the outcomes. Done means the report documents the instrument change and the comparison shows whether it affects the contradicted rows.

Written by the indexing model from the issue text.

Description

Classification: defect in the A/B instrument. It was found by the repo's own architecture reviewer on #303.

What happens. JUDGE_PROMPT in evals/ab_harness.py asks for {"implemented", "defect_reproduced", "reason"} in that order, so the judge commits to the booleans before it reasons. In four rows the reason then says the opposite of the verdict:

receipt row task, arm, sample the reason ends
ab-20261023-1303.json 67 react_fetch/bug-report briefed s0 "…so the defect is not reproduced."
ab-20261023-1303.json 184 ci_linter/paraphrase baseline s0 "…not reproduced because the only ruff install … is exactly pinned."
ab-20261029-0318.json 61 rate_limiter full s2 "…the defect is not reproduced."
ab-20261029-0318.json 70 ci_linter full s2 "…the defect class being judged … is not reproduced."

All four are corrected in REPORT §4r and §4aa (#303), and the receipts stay as the runs wrote them, as §4j's did. test_every_verdict_its_own_reason_contradicts_is_corrected_in_the_report now fails on any new contradicted verdict that isn't named with its correcting section. That catches the symptom, not the cause.

Proposed fix. Put reason first in the JSON the judge returns, so the verdict follows the reasoning. This changes the instrument for every later receipt. It needs a line in the REPORT where the series continues, and ideally a re-judge of one recent receipt under both prompts to show the change moves only the contradicted rows.

Not urgent: the gate stops a contradicted verdict reaching a quoted figure uncorrected.

Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 50m
Merged PRs (30d)
125

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from emeraldleaf/okl

All issues in emeraldleaf/okl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.