Eval scoring combines unrelated candidate fields
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 54/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- testing
Research direction
Start with the scoring logic at src/evals/eval-scoring.ts lines 95, 1399, 1476, and 1591, then trace how candidate fields are matched and how disappeared negative sentinels are handled. Reproduce the in-memory scorer case described in the issue and check the independent field matching observation. Done means a candidate only satisfies an expectation when that same candidate matches all required fields, with negative sentinels adjudicated separately.
Written by the indexing model from the issue text.
Description
Problem: In an eval with a left-path low-security candidate and a right-path high-correctness candidate, scoring merges their fields and passes a required left/high-security expectation with the right failure title even though no candidate satisfies all fields.
File:line: src/evals/eval-scoring.ts:95,1399,1476,1591.
Evidence: In-memory scorer probe; @architect checked independent field matching. Adjudicate disappeared negative sentinels separately when tightening the matcher.
Suggested fix size: M.
Found during #44 (workstream C, challenged by @architect).
- Dominant language
- TypeScript
- Stars
- 8
- Forks
- 1
- Avg merge
- 5m
- Merged PRs (30d)
- 2
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from 0xPolygon/codegenie
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
All issues in 0xPolygon/codegenie
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
feature request needs triage
Difficulty 1/5 1-3 hours Newbie friendliness 69/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/symphony#91 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
solidjs/solid-docs#1716 ·
Maintainers usually reply within 1 day