SelfPrompting fuzzy scoring gives credit to empty outputs
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start in the SelfPrompting._run_tasking() implementation and inspect the existing fuzzy scoring tests or nearby regression tests. Verify that empty sampled and expected strings receive credit only through exact matching, while non-empty substring matches retain their current behavior; add coverage for all three cases.
Written by the indexing model from the issue text.
Description
Describe the bug
SelfPrompting._run_tasking() computes fuzzy correctness with substring checks:
fuzzy = 1 if tasker_output in sample["output"] or sample["output"] in tasker_output else 0
In Python, the empty string is a substring of every string. A tasker that returns "" therefore receives fuzzy = 1 for any non-empty expected answer, even though it produced no answer at all. Likewise, an empty expected answer would fuzzy-match any model output.
Expected behavior
Empty strings should only be handled by exact matching; they should not receive fuzzy substring credit. Non-empty substring matches should retain the current behavior.
Proposed fix
Require both the sampled and expected strings to be non-empty before applying the substring-based fuzzy match, and add regression coverage for empty sampled/expected strings plus a normal non-empty fuzzy match.
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/evals
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100