MMMU multiple-choice scoring accepts answer prefixes and earlier discarded answers
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start at the MMMU.eval_sample() entry point and inspect the existing multiple-choice scoring and random-guess fallback. Add regression coverage for exact answers, prefix false positives, and multiple markers; done means the final valid marker is compared exactly while responses without a marker retain the existing fallback.
Written by the indexing model from the issue text.
Description
Describe the bug
MMMU.eval_sample() currently scores multiple-choice responses with a raw substring search:
match = sampled.find(f"ANSWER: {correct_answer}") != -1
This can mark incorrect responses as correct. For example, when the correct option is A, ANSWER: AB contains ANSWER: A and is accepted. Likewise, a response such as ANSWER: A ... actually, ANSWER: B is marked correct for A even though the model's final explicit answer is B.
Expected behavior
For multiple-choice MMMU samples, scoring should use the model's final valid ANSWER: X marker, where X is exactly one of A, B, C, or D. Prefixes such as ANSWER: AB should not count as option A.
The existing random-guess fallback when the model emits no ANSWER marker should remain unchanged.
Proposed fix
Extract valid multiple-choice answer markers with a boundary-aware helper, use the last valid marker as the final answer, and compare it exactly to the shuffled correct option. Add regression coverage for exact answers, prefix false positives, and multiple answer markers.
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from openai/evals
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100