exact_match: ignore_punctuation only removes ASCII punctuation

Open Beginner friendly
  • Is something wrong?
#836 0 comments 0 reactions 0 assignees View on GitHub

@reginaldalfret is already working on this.

Since Oct 22, 2026.

  • #837 by @reginaldalfret — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
75/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
machine-learning

Research direction

Start with the exact_match metric implementation and its tests, locating where ignore_punctuation uses string.punctuation. Check how punctuation is removed and add coverage for the Arabic comma and guillemets alongside the ASCII example in the issue. Done when Unicode punctuation is stripped and the examples score as expected.

Written by the indexing model from the issue text.

Description

ignore_punctuation=True is documented as removing all punctuation, but it uses string.punctuation, so non-ASCII punctuation like the Arabic comma ، or «» is kept and correct answers score 0. Stripping Unicode punctuation (categories P*) would fix it.

import evaluate
em = evaluate.load("exact_match")
em.compute(predictions=["Cairo,"], references=["Cairo"], ignore_punctuation=True)      # 1.0
em.compute(predictions=["القاهرة،"], references=["القاهرة"], ignore_punctuation=True)  # 0.0
Dominant language
Python
Stars
2.5k
Forks
380
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/evaluate

All issues in huggingface/evaluate

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.