exact_match scores two different answers as a perfect match when normalization erases both to the same residue

Open Beginner friendly
  • Is something wrong?
#822 0 comments 0 reactions 0 assignees View on GitHub

@MahendraMula is already working on this.

Since Sep 23, 2026.

  • #823 by @MahendraMula — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
machine-learning, testing

Research direction

The issue is in metrics/exact_match/exact_match.py, specifically the _compute function. Start by reading the function to understand how predictions and references are normalized. Run the provided test cases to reproduce the bug. The fix involves adding a check after normalization to handle cases where both strings become empty. Also, update the docstring at line 40 and add a test for the erase-to-identical case.

Written by the indexing model from the issue text.

Description

metrics/exact_match/exact_match.py, _compute (HEAD cae2ac3): regexes_to_ignore, ignore_case, ignore_punctuation and ignore_numbers are applied to predictions and references symmetrically, then

score_list = predictions == references
return {"exact_match": np.mean(score_list)}

There is no check that anything survived normalization. When it removes the whole discriminating content from both sides, the residue is identical — usually "" — and the row scores 1.0.

Measured

_compute copied verbatim (it is pure numpy + re + string; no evaluate import needed), controls in both directions:

ignore_numbers=True        pred='42'          ref='17'          -> 1.0   <-- different answers, perfect match
ignore_numbers=True        pred='1999'        ref='2024'        -> 1.0
ignore_punctuation=True    pred='?'           ref='!'           -> 1.0
regexes_to_ignore=[r'\d+'] pred='Answer: 42'  ref='Answer: 17'  -> 1.0

no flags                   pred='42'          ref='17'          -> 0.0   (control: correctly wrong)
no flags                   pred='42'          ref='42'          -> 1.0   (control: correctly right)
ignore_case=True           pred='Paris'       ref='paris'       -> 1.0   (control: the intended use)
ignore_numbers=True        pred='Chapter 3'   ref='Chapter 4'   -> 1.0   (control: intended — 'Chapter ' survives on both sides)

The last control is the point: the options behave as designed when text survives. The failure is specifically total erasure — a numeric-answer task with ignore_numbers=True, or a regexes_to_ignore pattern that happens to consume the answer, reports 100% on every row.

What the docs say

The docstring describes each flag but does not mention that normalization can consume the entire string. Line 40 also reads "ignore_numbers: If true, removes all punctuation before…" — a copy-paste of the ignore_punctuation line. I could not find a test that exercises ignore_numbers, ignore_punctuation, or regexes_to_ignore.

Consequence

A benchmark that normalizes away digits or punctuation before comparing — a common configuration for numeric or short-answer tasks — can score a model that is wrong on every row as perfect, and nothing in the result says the comparison was between two empty strings.

Suggested fix

Either of:

  1. After normalization, treat a row where both sides are empty (or where a side was non-empty before and empty after) as a non-match, or as excluded with a count reported alongside the score.
  2. Emit a warning when normalization empties any string, so the 1.0 is at least visibly suspect.

Plus a one-line docstring fix at line 40 and a test for the erase-to-identical case. Happy to open the PR.

Dominant language
Python
Stars
2.5k
Forks
380
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/evaluate

All issues in huggingface/evaluate

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.