Explanation contract is unenforced — set the v2 scoring policy
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- python
- Domain
- ai, testing-qa
Research direction
Start with the August 2026 harness audit and issue #139's v2 freeze context. Review the current scoring behavior for blank explanations and compare it with the documented parse-failure policy. Done means the v2 methodology explicitly selects and documents one explanation-scoring policy, including how explanation compliance is reported.
Written by the indexing model from the issue text.
Description
From the August 2026 harness audit (policy decision needed before the #139 v2 freeze).
Symptom
The response contract requires a short explanation per output, but the requirement is not enforced at scoring time: blank-explanation answers score normally.
Evidence (verified)
- 61 blank-explanation rows accepted on the current board: grok-4.3 = 56, kimi-k2.6 = 4, haiku-4.5 = 1.
- If blanks scored as misses, grok-4.3 would drop 2.574 pts (~7 ranks). No other model moves materially.
Decision for v2
Two coherent options:
- Score on numbers only (recommended): keep the explanation request in the prompt, publish explanation-compliance as a per-model column, but don't gate the accuracy score on prose compliance. Scoring prose mixes constructs into a numeric-accuracy headline.
- Enforce as miss, stated up front alongside the parse-failure policy (like Sonnet 5's 56/1,984 unparsed-as-miss under auto).
Either way the v2 methodology must state the policy explicitly; today's behavior is option 1 by accident, undocumented.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 11h 11m
- Merged PRs (30d)
- 22
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PolicyEngine/policybench
-
v2 prompt and data: program-specific disability facts, integer inputs, and the take-up boundaryOpen
Difficulty 5/5 Over a week Newbie friendliness 38/100
PolicyEngine/policybench#165 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 52/100
PolicyEngine/policybench#140 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
PolicyEngine/policybench#139 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
PolicyEngine/policybench#129 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
PolicyEngine/policybench#118 ·
Maintainers usually reply within 1 day
All issues in PolicyEngine/policybench
Similar issues
-
Path subtraction removes repeated suffix characters instead of one suffixPossibly taken @Gwerdonatus claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
joke2k/django-environ#671 ·
-
good first issue help wanted windows
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
-
test_pgbouncer deploys the server under test from Charmhub instead of the local build (track/1.31)Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
canonical/temporal-k8s-operator#195 · 1 comment ·
Maintainers usually reply within 4 days
-
`burr` fails to start when the Python path has a space in itPossibly taken @enekoreto claimed this today. Openstatus/needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 5 days
-
device support request
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
zigpy/zha-device-handlers#5461 ·
Maintainers usually reply within 4 days