[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning, testing-qa
Research direction
Read src/lighteval/tasks/tasks/arc_agi_2.py and trace how the prompt formats grids and supplies gold answers; then inspect Metrics.exact_match and its tests. Add coverage for the row-per-line answer and confirm both that format and the existing one-line list score 1.
Written by the indexing model from the issue text.
Description
Describe the bug
arc_agi_2_prompt (src/lighteval/tasks/tasks/arc_agi_2.py) shows every grid one row per line and asks the model to "Respond in the format of the training output examples", but the gold is str(output), the one-line list [[0, 0], [0, 1]]. Metrics.exact_match compares stripped strings, so on the non-Inspect backends (accelerate, vllm, endpoint) a correct grid written the way the prompt shows it scores 0. The Inspect path is fine because exact() ignores punctuation.
To Reproduce
from lighteval.metrics.metrics import Metrics
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.tasks.arc_agi_2 import arc_agi_2_prompt
record = {"fewshots": [{"input": [[1, 0], [0, 0]], "output": [[0, 1], [0, 0]]}],
"question": [{"input": [[0, 0], [1, 0]], "output": [[0, 0], [0, 1]]}]}
doc = arc_agi_2_prompt(record, "arc_agi_2") # example OUTPUT in the query: "[0, 1]\n[0, 0]"
print(doc.get_golds()) # ['[[0, 0], [0, 1]]']
em = Metrics.exact_match.value.sample_level_fn
for answer in ["[0, 0]\n[0, 1]", "[[0, 0], [0, 1]]"]:
print(repr(answer), em.compute(doc=doc, model_response=ModelResponse(text=[answer])))
# '[0, 0]\n[0, 1]' 0
# '[[0, 0], [0, 1]]' 1
Expected behavior
A correct grid in the prompt's row-per-line format should score 1 (the one-line list can keep scoring 1).
Version info
lighteval main (b97a653), Python 3.11, Ubuntu (WSL2). Happy to send a PR with a unit test.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 612
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/lighteval
-
[BUG] vllm_main_tests.yaml: continue-on-error hides a canary 23 scheduled runs deep in failurePossibly taken @voropaevv claimed this 9 days ago. Open
Difficulty 2/5 Under an hour Newbie friendliness 76/100
huggingface/lighteval#1486 ·
Maintainers usually reply within 1 day
-
[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literalsPossibly taken @inchang-ing claimed this 15 days ago. Openbug
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
huggingface/lighteval#1471 ·
Maintainers usually reply within 1 day
-
[BUG] Stderr of `_average` and `all` rows is the mean of the subtask stderrs (about 7x too large for MMLU)Possibly taken @Andreas-Hum claimed this 17 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
huggingface/lighteval#1466 ·
Maintainers usually reply within 1 day
-
[BUG] Reasoning tag removal fails when <think> is part of the input promptPossibly taken @selimfedakar claimed this 21 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/lighteval#1453 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] TruthfulQA MC2 drops true answers after a gap in gold indexesPossibly taken @salma2vec claimed this 29 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/lighteval#1442 ·
Maintainers usually reply within 1 day
All issues in huggingface/lighteval
Similar issues
-
automated endpoint-dead needs-curation
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
almanac-data/climate-almanac#138 ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
OceanDataTools/openrvdas#760 ·
Maintainers usually reply within 1 day
-
level/subtask type/enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
wazuh/wazuh-virtual-machines#1091 ·
Maintainers usually reply within 1 day
-
Python: Bug: ChatMessageContent.from_element drops AudioContent and BinaryContent into raw TextContentPossibly taken @halcyon-past claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
microsoft/semantic-kernel#14824 · 1 comment ·
Maintainers usually reply within 3 days
-
help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
stack314/django-neolith#142 · 1 comment ·
Maintainers usually reply within 1 day