[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literals
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 74/100
Research direction
The empty entries are in src/lighteval/tasks/templates/utils/translation_literals.py, which the AttributeError message names. Run the reproduction snippet from the issue, then compare the Latvian, Lithuanian, Maltese and Slovenian blocks with a filled European entry such as Estonian. Done when TRANSLATION_LITERALS[Language.X].answer resolves for all four languages and belebele_{lvs,lit,mlt,slv}_Latn_cf report their document counts.
Written by the indexing model from the issue text.
Description
Describe the bug
belebele_{lvs,lit,mlt,slv}_Latn_cf can't be built, because TRANSLATION_LITERALS has empty entries for Latvian, Lithuanian, Maltese and Slovenian. Building the documents fails before any model is loaded:
AttributeError: Translation for 'answer' is needed for Language.LATVIAN. Please provide its implementation by editing the 'src/lighteval/tasks/templates/utils/translation_literals.py'
The other European Belebele cf tasks we run (Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Polish, Portuguese, Romanian, Slovak, Spanish, Swedish, Norwegian, Basque, Catalan) build fine. The entries have been empty in every release from v0.7.0 to v0.13.0.
To Reproduce
from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.registry import Registry
for lang in ("eng_Latn", "lvs_Latn", "lit_Latn", "mlt_Latn", "slv_Latn"):
name = f"belebele_{lang}_cf"
tasks = Registry(tasks=f"{name}|0", load_multilingual=True).load_tasks()
LightevalTask.load_datasets(tasks)
(task,) = tasks.values()
try:
print(name, len(task.eval_docs()), "docs")
except AttributeError as e:
print(name, "failed:", e)
Output: belebele_eng_Latn_cf 900 docs, then the AttributeError above for Latvian, Lithuanian, Maltese and Slovenian. The cause is visible directly:
from lighteval.tasks.templates.utils.translation_literals import TRANSLATION_LITERALS
from lighteval.utils.language import Language
TRANSLATION_LITERALS[Language.LATVIAN].answer # AttributeError; same for LITHUANIAN, MALTESE, SLOVENIAN
Expected behavior
The four tasks build and run like the other Belebele cf languages. That needs translation literals for Latvian, Lithuanian, Maltese and Slovenian in translation_literals.py, at least answer, which the Belebele cf prompt uses.
Version info
- lighteval 0.13.1.dev0, installed from git commit 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7. The four entries are still empty on
main(b97a6533). - Python 3.12.3, Ubuntu 24.04.4 (Apptainer container)
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 612
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/lighteval
-
[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)Possibly taken @BahaaMattar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
huggingface/lighteval#1510 ·
Maintainers usually reply within 1 day
-
[BUG] vllm_main_tests.yaml: continue-on-error hides a canary 23 scheduled runs deep in failurePossibly taken @voropaevv claimed this 9 days ago. Open
Difficulty 2/5 Under an hour Newbie friendliness 76/100
huggingface/lighteval#1486 ·
Maintainers usually reply within 1 day
-
[BUG] Stderr of `_average` and `all` rows is the mean of the subtask stderrs (about 7x too large for MMLU)Possibly taken @Andreas-Hum claimed this 17 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
huggingface/lighteval#1466 ·
Maintainers usually reply within 1 day
-
[BUG] Reasoning tag removal fails when <think> is part of the input promptPossibly taken @selimfedakar claimed this 21 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/lighteval#1453 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] TruthfulQA MC2 drops true answers after a gap in gold indexesPossibly taken @salma2vec claimed this 29 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/lighteval#1442 ·
Maintainers usually reply within 1 day
All issues in huggingface/lighteval
Similar issues
-
integration: media_player
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
home-assistant/core#193344 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
daniel-ospina/tortoise#13607 ·
Maintainers usually reply within 1 day
-
documentation priority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day