Repository Issues

huggingface/lighteval

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

View on GitHub
Stars
 (2,496 stars)
Forks
 (514 forks)
Indexed issues
 (23 indexed issues)
open beginner issues
 (23 open beginner issues)
Latest indexed
Aug 14, 2026
Last GitHub push
Jun 29, 2026
License
MIT License
Contributing guide
No contributing guide
Code of conduct
No code of conduct
Dominant language
Python
PR merge metrics
 (PR metrics pending)
Beginner labels
good first issuehelp wanted

Issues

23 open indexed issues

Open
[EVAL] Long Horizon Execution
good first issuehelp wantednew-task
Why recommendedHas a beginner-friendly label
Has a beginner-friendly label

huggingface / lighteval · #1056 · Nov 13, 2025 · Python · 2,496 stars

7 comments0 reactions1 assignee
Open
[FT] Add tests for nanotron
featuregood first issuehelp wantedignore-for-release
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #765 · May 21, 2025 · Python · 2,496 stars

3 comments0 reactions0 assignees
Open
[EVAL] TauBench:
help wantednew-task
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #741 · May 19, 2025 · Python · 2,496 stars

1 comment0 reactions0 assignees
Open
[BUG] Optimize tokenization
buggood first issuehelp wantedscience-team
Why recommendedHas a beginner-friendly label
Has a beginner-friendly label

huggingface / lighteval · #732 · May 15, 2025 · Python · 2,496 stars

1 comment0 reactions1 assignee
Open
[EVAL] HELMET: long context evals
help wantednew-task
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #731 · May 15, 2025 · Python · 2,496 stars

2 comments0 reactions0 assignees
Open
[EVAL] SWEBENCH multilingual
help wantednew-task
Why recommendedNo assignee yet · No comments yet
No assignee yetNo comments yetHas a beginner-friendly label

huggingface / lighteval · #730 · May 15, 2025 · Python · 2,496 stars

0 comments0 reactions0 assignees
Open
[EVAL] Adding PHARE
good first issuehelp wantednew-task
Why recommendedHas a beginner-friendly label
Has a beginner-friendly label

huggingface / lighteval · #696 · Apr 30, 2025 · Python · 2,496 stars

9 comments0 reactions1 assignee
Open
[FT] Improve Documentation and Examples
documentationfeaturegood first issuehelp wanted
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #682 · Apr 21, 2025 · Python · 2,496 stars

6 comments1 reaction0 assignees
Open
[EVAL] Big-Bench Extra Hard (BBEH)
good first issuehelp wantednew-taskscience-team
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #600 · Mar 3, 2025 · Python · 2,496 stars

3 comments0 reactions0 assignees
Open
[EVAL] Add TUMLU benchmark
good first issuehelp wantednew-task
Why recommendedNo assignee yet · Has a beginner-friendly label
No assignee yetHas a beginner-friendly label

huggingface / lighteval · #577 · Feb 19, 2025 · Python · 2,496 stars

10 comments0 reactions0 assignees