EvolvingLMMs-Lab/lmms-eval

View on GitHub

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Stars: 4.4k
Forks: 656
Open beginner issues: 3
Indexed issues: 24
Avg merge: 2d 6h
Merged PRs (30d): 40

Dominant language: Python
License: No license data
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 20, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

24 open issues indexed

VideoMME score is substantially higher than the reported LLaVA-OneVision-7B result with 32 frames Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 50/100

EvolvingLMMs-Lab/lmms-eval#1541 · 4 comments · Opened Sep 19, 2026
mathvision: find_math_answer keeps the FIRST \\boxed answer when a model retracts it and boxes a corrected one Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

EvolvingLMMs-Lab/lmms-eval#1511 · 1 comment · Opened Sep 4, 2026
mathvision: find_math_answer deletes the \text command name but keeps its braces — \boxed{\text{Sunday}} is scored wrong against gold Sunday Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

EvolvingLMMs-Lab/lmms-eval#1509 · 1 comment · Opened Sep 4, 2026
mathvision: _fix_fracs is gated on "sqrt" — \frac12-style shorthand never normalizes, silently failing is_equal for sqrt-free answers Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

EvolvingLMMs-Lab/lmms-eval#1507 · 1 comment · Opened Sep 4, 2026
mathvision: _fix_sqrt corrupts indexed roots (\sqrt[3]{8} → \sqrt{[}3]{8}), silently failing the latex2sympy equivalence path Open

Difficulty 2/5 1-3 hours Newbie friendliness 86/100

EvolvingLMMs-Lab/lmms-eval#1505 · 2 comments · Opened Sep 4, 2026
Verify evals on Papers with Code Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

EvolvingLMMs-Lab/lmms-eval#1483 · Opened Aug 28, 2026
Group-level post-processing hook for custom result aggregation Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 38/100

EvolvingLMMs-Lab/lmms-eval#1406 · Opened Jul 22, 2026
`qwen2_5_vl` video metadata fix in #1269 does not affect `second_per_grid_ts`; effective workaround is to pass fps, but fps > 2 can break temporal position ids Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 42/100

EvolvingLMMs-Lab/lmms-eval#1393 · 1 comment · Opened Jul 14, 2026
Add Gemma4 and Qwen3.6 support Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

EvolvingLMMs-Lab/lmms-eval#1315 · 1 comment · 1 reaction · Opened Apr 30, 2026
Add llama.cpp support Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#1312 · Opened Apr 27, 2026
qwen3_5是文本图像多模态模型，但目前只能测评文本 Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

EvolvingLMMs-Lab/lmms-eval#1300 · 3 comments · 1 reaction · Opened Apr 15, 2026
Should worldqa benchmark contain audio? Open

Difficulty 3/5 1-2 days Newbie friendliness 40/100

EvolvingLMMs-Lab/lmms-eval#1288 · Opened Apr 8, 2026
你好会支持OmniDocBench1.5评测 Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

EvolvingLMMs-Lab/lmms-eval#1287 · Opened Apr 7, 2026
uv sync fails on IBM Power 10 linux ppc64le system Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 52/100

EvolvingLMMs-Lab/lmms-eval#1283 · Opened Apr 2, 2026
Decouple reasoning and evaluation Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

EvolvingLMMs-Lab/lmms-eval#1259 · Opened Mar 19, 2026
Integrate with EgoLifeQA Benchmark Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#1214 · Opened Feb 28, 2026
Async evaluation for HF models (async_hf_model backend) Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 32/100

EvolvingLMMs-Lab/lmms-eval#1126 · 3 comments · Opened Feb 21, 2026
Issues about MMVP Dataset Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#1018 · 1 comment · Opened Jan 22, 2026
When I tested Qwen-2.5-vl-7B on the Videomme dataset, ValueErrorr: The decoder prompt (length 163270) is longer Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#988 · 5 comments · Opened Jan 12, 2026
用vllm和不用vllm结果差的很多 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

EvolvingLMMs-Lab/lmms-eval#921 · 2 comments · Opened Dec 3, 2025
The LLM Judge function takes too long, resulting in low GPU utilization. Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

EvolvingLMMs-Lab/lmms-eval#908 · 3 comments · Opened Nov 20, 2025
Error while runnig lava-onevision-qwen2-7b example Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#860 · 6 comments · Opened Oct 17, 2025
Performance gap between python and vLLM backends using the same Qwen2-VL-2B-Instruct model on ChartQA task Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

EvolvingLMMs-Lab/lmms-eval#698 · 5 comments · 4 reactions · Opened May 30, 2025
internvl 2.5 8b 和qwen2.5vl的结果 Open

discussion

Difficulty 4/5 3-5 days Newbie friendliness 25/100

EvolvingLMMs-Lab/lmms-eval#603 · 13 comments · Opened Mar 27, 2025

EvolvingLMMs-Lab/lmms-eval

Get new issues in your inbox