Eval scoring combines unrelated candidate fields

Open
#48 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
54/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
testing

Research direction

Start with the scoring logic at src/evals/eval-scoring.ts lines 95, 1399, 1476, and 1591, then trace how candidate fields are matched and how disappeared negative sentinels are handled. Reproduce the in-memory scorer case described in the issue and check the independent field matching observation. Done means a candidate only satisfies an expectation when that same candidate matches all required fields, with negative sentinels adjudicated separately.

Written by the indexing model from the issue text.

Description

bug

Problem: In an eval with a left-path low-security candidate and a right-path high-correctness candidate, scoring merges their fields and passes a required left/high-security expectation with the right failure title even though no candidate satisfies all fields.
File:line: src/evals/eval-scoring.ts:95,1399,1476,1591.
Evidence: In-memory scorer probe; @architect checked independent field matching. Adjudicate disappeared negative sentinels separately when tightening the matcher.
Suggested fix size: M.
Found during #44 (workstream C, challenged by @architect).

Dominant language
TypeScript
Stars
8
Forks
1
Avg merge
5m
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from 0xPolygon/codegenie

All issues in 0xPolygon/codegenie

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.