[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literals

Open Beginner friendly
#1,471 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@inchang-ing is already working on this.

Since Nov 3, 2026.

  • #1473 by @inchang-ing — open

Assessment

Difficulty
1/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
tooling

Research direction

The empty entries are in src/lighteval/tasks/templates/utils/translation_literals.py, which the AttributeError message names. Run the reproduction snippet from the issue, then compare the Latvian, Lithuanian, Maltese and Slovenian blocks with a filled European entry such as Estonian. Done when TRANSLATION_LITERALS[Language.X].answer resolves for all four languages and belebele_{lvs,lit,mlt,slv}_Latn_cf report their document counts.

Written by the indexing model from the issue text.

Description

bug

Describe the bug

belebele_{lvs,lit,mlt,slv}_Latn_cf can't be built, because TRANSLATION_LITERALS has empty entries for Latvian, Lithuanian, Maltese and Slovenian. Building the documents fails before any model is loaded:

AttributeError: Translation for 'answer' is needed for Language.LATVIAN. Please provide its implementation by editing the 'src/lighteval/tasks/templates/utils/translation_literals.py'

The other European Belebele cf tasks we run (Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Polish, Portuguese, Romanian, Slovak, Spanish, Swedish, Norwegian, Basque, Catalan) build fine. The entries have been empty in every release from v0.7.0 to v0.13.0.

To Reproduce

from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.registry import Registry

for lang in ("eng_Latn", "lvs_Latn", "lit_Latn", "mlt_Latn", "slv_Latn"):
    name = f"belebele_{lang}_cf"
    tasks = Registry(tasks=f"{name}|0", load_multilingual=True).load_tasks()
    LightevalTask.load_datasets(tasks)
    (task,) = tasks.values()
    try:
        print(name, len(task.eval_docs()), "docs")
    except AttributeError as e:
        print(name, "failed:", e)

Output: belebele_eng_Latn_cf 900 docs, then the AttributeError above for Latvian, Lithuanian, Maltese and Slovenian. The cause is visible directly:

from lighteval.tasks.templates.utils.translation_literals import TRANSLATION_LITERALS
from lighteval.utils.language import Language

TRANSLATION_LITERALS[Language.LATVIAN].answer  # AttributeError; same for LITHUANIAN, MALTESE, SLOVENIAN

Expected behavior

The four tasks build and run like the other Belebele cf languages. That needs translation literals for Latvian, Lithuanian, Maltese and Slovenian in translation_literals.py, at least answer, which the Belebele cf prompt uses.

Version info

  • lighteval 0.13.1.dev0, installed from git commit 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7. The four entries are still empty on main (b97a6533).
  • Python 3.12.3, Ubuntu 24.04.4 (Apptainer container)
Dominant language
Python
Stars
2.6k
Forks
612
Avg merge
2d 3h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/lighteval

All issues in huggingface/lighteval

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.