Failed review runs pass negative evals
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- testing-qa
Research direction
Read src/evals/eval-runner.ts at lines 487 and 576, src/evals/eval-scoring.ts at line 69, and src/cli/main.ts at line 50; compare how live evals handle failed run health with the CLI. The probe described in the issue uses a failed run.json outcome, exit code 1, and zero findings. Preserve retained findings and report failed samples separately; the aggregate policy for mixed valid and failed draws still needs a decision.
Written by the indexing model from the issue text.
Description
Problem: Both live eval paths ignore review health. A well-formed zero-finding negative eval with run.json outcome failed and exit code 1 scores as pass. Preserve retained findings and report the failed sample separately; aggregate policy for mixed valid and failed draws needs a decision.
File:line: src/evals/eval-runner.ts:487,576; src/evals/eval-scoring.ts:69; src/cli/main.ts:50.
Evidence: Probe with failed run health and zero findings; CLI honors failed health.
Suggested fix size: S.
Found during #44 (workstream C, challenged by @architect).
- Dominant language
- TypeScript
- Stars
- 8
- Forks
- 1
- Avg merge
- 5m
- Merged PRs (30d)
- 2
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from 0xPolygon/codegenie
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
All issues in 0xPolygon/codegenie
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
feature request needs triage
Difficulty 1/5 1-3 hours Newbie friendliness 69/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/symphony#91 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
solidjs/solid-docs#1716 ·
Maintainers usually reply within 1 day