[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)

Open Beginner friendly
#1,510 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@BahaaMattar is already working on this.

Since Nov 18, 2026.

  • #1511 by @BahaaMattar — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Read src/lighteval/tasks/tasks/arc_agi_2.py and trace how the prompt formats grids and supplies gold answers; then inspect Metrics.exact_match and its tests. Add coverage for the row-per-line answer and confirm both that format and the existing one-line list score 1.

Written by the indexing model from the issue text.

Description

Describe the bug

arc_agi_2_prompt (src/lighteval/tasks/tasks/arc_agi_2.py) shows every grid one row per line and asks the model to "Respond in the format of the training output examples", but the gold is str(output), the one-line list [[0, 0], [0, 1]]. Metrics.exact_match compares stripped strings, so on the non-Inspect backends (accelerate, vllm, endpoint) a correct grid written the way the prompt shows it scores 0. The Inspect path is fine because exact() ignores punctuation.

To Reproduce

from lighteval.metrics.metrics import Metrics
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.tasks.arc_agi_2 import arc_agi_2_prompt

record = {"fewshots": [{"input": [[1, 0], [0, 0]], "output": [[0, 1], [0, 0]]}],
          "question": [{"input": [[0, 0], [1, 0]], "output": [[0, 0], [0, 1]]}]}
doc = arc_agi_2_prompt(record, "arc_agi_2")   # example OUTPUT in the query: "[0, 1]\n[0, 0]"
print(doc.get_golds())                         # ['[[0, 0], [0, 1]]']
em = Metrics.exact_match.value.sample_level_fn
for answer in ["[0, 0]\n[0, 1]", "[[0, 0], [0, 1]]"]:
    print(repr(answer), em.compute(doc=doc, model_response=ModelResponse(text=[answer])))
# '[0, 0]\n[0, 1]' 0
# '[[0, 0], [0, 1]]' 1

Expected behavior

A correct grid in the prompt's row-per-line format should score 1 (the one-line list can keep scoring 1).

Version info

lighteval main (b97a653), Python 3.11, Ubuntu (WSL2). Happy to send a PR with a unit test.

Dominant language
Python
Stars
2.6k
Forks
612
Avg merge
2d 3h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/lighteval

All issues in huggingface/lighteval

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.