eval --compare collides on identical commands (intent twins)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- typescript
- Domain
- testing-qa, tooling
Research direction
Start in eval/run.ts at the previous-report map around line 1485, then reproduce with --corpus intent --compare using eval/corpus/intent.jsonl. Make comparison keys distinguish identical commands by evidence and label or a row id, reject duplicate keys, and verify that the comparison no longer reports twins against the wrong row.
Written by the indexing model from the issue text.
Description
eval/run.ts --compare keys the previous report by command string alone (new Map(previous.map(o => [o.command, o])), eval/run.ts:1485). When two rows share a command, the last one in the old report wins, and every current row with that command is diffed against it.
eval/corpus/intent.jsonl (#76) has 12+ twin pairs with the same command and opposite labels, so the first --corpus intent --compare run will print FIXED/REGRESSION lines against the wrong row.
Fix: key on command plus the evidence and label (or a row id), and fail loudly if two rows still share a key.
Found by the review gate on #76. It predates that PR: --corpus heldout already repeats 20 commands 25 times each.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 1
- Avg merge
- 1h 20m
- Merged PRs (30d)
- 32
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from STRML/omp-classifier
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
STRML/omp-classifier#81 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
STRML/omp-classifier#78 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
STRML/omp-classifier#84 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
STRML/omp-classifier#82 ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
STRML/omp-classifier#73 ·
All issues in STRML/omp-classifier
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug clawsweeper:linked-pr-open clawsweeper:needs-live-repro clawsweeper:no-new-fix-pr impact:message-loss issue-rating: 🐚 platinum hermit P2 regression
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
danielmiessler/LifeOS#2218 ·