[BUG] Stderr of `_average` and `all` rows is the mean of the subtask stderrs (about 7x too large for MMLU)
Maintainers usually reply within 1 day
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 70/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning, tooling
Research direction
Start in MetricsLogger.aggregate in src/lighteval/logging/info_loggers.py, where the _average group rows and the all row sum every metric key, including *_stderr. Run the repro script in the issue to confirm the stderr equals the mean of the subtask stderrs. Done means _stderr keys combine as sqrt(sum of squares) / G while point estimates stay the same, which the issue's fix_check script asserts.
Written by the indexing model from the issue text.
Description
Describe the bug
For task families with several subtasks, MetricsLogger.aggregate builds a group row (e.g. mmlu:_average|5) and the all row by averaging every key of the subtask results, including the <metric>_stderr keys. The stderr shown for these rows is therefore the arithmetic mean of the subtask stderrs, not the standard error of the averaged score.
The averaged score is the unweighted mean of G subtask scores computed on disjoint question sets, so its standard error is sqrt(sum_i se_i^2) / G. The mean of the stderrs is larger by the factor sum_i se_i / sqrt(sum_i se_i^2), which lies between 1 and sqrt(G). For the 57 MMLU subjects it is about 7.3 (sqrt(57) = 7.55): in the repro below the results table shows 0.6077 ± 0.0345 (95% interval ±6.8 points) while the standard error of that average is 0.0047 (±0.9 points). Every multi-subtask family (mmlu:*, bigbench_hard:*, bigbench:*, ...) is affected in its _average row and in all.
To Reproduce
The script below feeds synthetic per-question scores for the 57 real MMLU task configs (with the MMLU test-set sizes) through MetricsLogger.log / MetricsLogger.aggregate and prints the rows with make_results_table. No model or dataset download is needed.
import numpy as np
from lighteval.logging.info_loggers import MetricsLogger
from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.tasks.mmlu import TASKS_TABLE
from lighteval.utils.utils import make_results_table
# MMLU test-set sizes of the 57 subjects (alphabetical order, 14042 questions)
N = np.array([100, 135, 152, 100, 265, 144, 100, 100, 100, 173, 102, 100, 235, 114, 145, 378, 126, 100, 310,
203, 100, 165, 198, 193, 390, 270, 238, 151, 545, 216, 204, 237, 223, 131, 121, 108, 163, 112,
103, 234, 100, 783, 346, 895, 306, 311, 324, 282, 1534, 272, 612, 110, 245, 201, 100, 166, 171])
tasks = {cfg.full_name: LightevalTask(cfg) for cfg in TASKS_TABLE} # the 57 real MMLU tasks (no data download)
names = sorted(tasks)
assert len(names) == len(N) == 57 and N.sum() == 14042
rng = np.random.default_rng(0)
p = rng.uniform(0.35, 0.85, size=57) # synthetic per-subject accuracy
logger = MetricsLogger()
for name, n, p_k in zip(names, N, p):
for correct in rng.random(n) < p_k: # synthetic per-question exact-match scores
logger.log(name, {"em": float(correct)})
logger.aggregate(task_dict=tasks, bootstrap_iters=1000)
res = logger.metric_aggregated
print(make_results_table({"results": {k: res[k] for k in ("all", "mmlu:_average|0")}, "versions": {}}))
se = np.array([res[k]["em_stderr"] for k in names])
expected = np.sqrt(np.sum(se**2)) / len(se) # SE of the mean of 57 independent subtask means
mc = (rng.binomial(N, p, size=(20000, 57)) / N).mean(axis=1).std() # SD of the score over fresh question draws
for row in ("mmlu:_average|0", "all"):
actual = res[row]["em_stderr"]
print(f"{row:15s} em_stderr: actual {actual:.5f} | expected sqrt(sum se_i^2)/G = {expected:.5f} "
f"(Monte Carlo SD {mc:.5f}) | actual/expected = {actual / expected:.2f}")
assert np.isclose(actual, se.mean()) # actual = arithmetic mean of the 57 subtask stderrs
assert actual / expected > 5 # inflated ~7x (the limit for equal subtask stderrs is sqrt(57) = 7.55)
Output:
| Task |Version|Metric|Value | |Stderr|
|---------------|-------|------|-----:|---|-----:|
|all | |em |0.6077|± |0.0345|
|mmlu:_average\|0| |em |0.6077|± |0.0345|
mmlu:_average|0 em_stderr: actual 0.03452 | expected sqrt(sum se_i^2)/G = 0.00475 (Monte Carlo SD 0.00473) | actual/expected = 7.27
all em_stderr: actual 0.03452 | expected sqrt(sum se_i^2)/G = 0.00475 (Monte Carlo SD 0.00473) | actual/expected = 7.27
The reported em_stderr of mmlu:_average|0 and all (0.03452) is exactly the arithmetic mean of the 57 subtask stderrs.
Expected behavior
The stderr of an averaged row should be the standard error of that average, sqrt(sum_i se_i^2) / G, which is also what a Monte Carlo over fresh question draws gives:
|all | |em |0.6077|± |0.0047|
|mmlu:_average\|0| |em |0.6077|± |0.0047|
lm-evaluation-harness, which metrics/utils/stderr.py is adapted from, uses this formula for unweighted group means (unweighted_mean_stderr, lm_eval/api/metrics.py#L643-L651 at v0.4.13).
Root cause
src/lighteval/logging/info_loggers.py#L374-L402 (v0.13.0, 7c1cd62):
- group rows, L393-L396:
metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks)is applied to every key, including<metric>_stderr; allrow, L385-L387 and L399-L400: the same sum-then-divide for every key.
The rows are then printed as Value ± Stderr by make_results_table. The code is unchanged on main (b97a653, same lines).
Proposed fix
For _stderr keys, add the squares and take the square root before dividing by the number of subtasks (independent errors add in quadrature). Point estimates are unchanged.
--- a/src/lighteval/logging/info_loggers.py
+++ b/src/lighteval/logging/info_loggers.py
@@ -383,7 +383,9 @@
task, fewshot = k.split("|")
grouped_tasks[f"{task.split(':')[0]}:_average|{fewshot}"].append(k)
for metric, value in metrics.items():
- suite_average[metric] = suite_average.get(metric, 0) + value
+ suite_average[metric] = suite_average.get(metric, 0) + (
+ value**2 if metric.endswith("_stderr") else value
+ )
suite_nb[metric] = suite_nb.get(metric, 0) + 1
# Compute average for sub groups
@@ -391,13 +393,18 @@
if len(list_of_subtasks) > 1:
metrics = list(self.metric_aggregated[list_of_subtasks[0]].keys())
self.metric_aggregated[average_task] = {
- metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks)
+ metric: (
+ sum(self.metric_aggregated[k][metric] ** 2 for k in list_of_subtasks) ** 0.5
+ if metric.endswith("_stderr")
+ else sum(self.metric_aggregated[k][metric] for k in list_of_subtasks)
+ )
+ / len(list_of_subtasks)
for metric in metrics
}
# Compute average for all
for metric, value in suite_average.items():
- suite_average[metric] = value / suite_nb[metric]
+ suite_average[metric] = (value**0.5 if metric.endswith("_stderr") else value) / suite_nb[metric]
self.metric_aggregated["all"] = suite_average
For all this assumes the tasks in the run are evaluated on different questions, which holds unless the same benchmark is included twice (e.g. at two few-shot settings).
Verifying the fix
fix_check.py applies the diff above to the installed MetricsLogger.aggregate and re-runs the same synthetic data:
fix_check.py
import inspect
import textwrap
import numpy as np
import lighteval.logging.info_loggers as info_loggers
from lighteval.logging.info_loggers import MetricsLogger
from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.tasks.mmlu import TASKS_TABLE
from lighteval.utils.utils import make_results_table
# Apply the proposed diff to the installed MetricsLogger.aggregate (stderrs of independent subtasks add in quadrature)
FIX = [
("suite_average[metric] = suite_average.get(metric, 0) + value",
'suite_average[metric] = suite_average.get(metric, 0) + (\n'
' value**2 if metric.endswith("_stderr") else value\n)'),
("metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks)",
"metric: (\n"
" sum(self.metric_aggregated[k][metric] ** 2 for k in list_of_subtasks) ** 0.5\n"
' if metric.endswith("_stderr")\n'
" else sum(self.metric_aggregated[k][metric] for k in list_of_subtasks)\n"
")\n/ len(list_of_subtasks)"),
("suite_average[metric] = value / suite_nb[metric]",
'suite_average[metric] = (value**0.5 if metric.endswith("_stderr") else value) / suite_nb[metric]'),
]
src = textwrap.dedent(inspect.getsource(MetricsLogger.aggregate))
for old, new in FIX:
assert src.count(old) == 1, old
src = src.replace(old, new)
namespace = {}
exec(src, vars(info_loggers), namespace)
MetricsLogger.aggregate = namespace["aggregate"]
# Same synthetic data as repro.py
N = np.array([100, 135, 152, 100, 265, 144, 100, 100, 100, 173, 102, 100, 235, 114, 145, 378, 126, 100, 310,
203, 100, 165, 198, 193, 390, 270, 238, 151, 545, 216, 204, 237, 223, 131, 121, 108, 163, 112,
103, 234, 100, 783, 346, 895, 306, 311, 324, 282, 1534, 272, 612, 110, 245, 201, 100, 166, 171])
tasks = {cfg.full_name: LightevalTask(cfg) for cfg in TASKS_TABLE}
names = sorted(tasks)
rng = np.random.default_rng(0)
p = rng.uniform(0.35, 0.85, size=57)
logger = MetricsLogger()
for name, n, p_k in zip(names, N, p):
for correct in rng.random(n) < p_k:
logger.log(name, {"em": float(correct)})
logger.aggregate(task_dict=tasks, bootstrap_iters=1000)
res = logger.metric_aggregated
print(make_results_table({"results": {k: res[k] for k in ("all", "mmlu:_average|0")}, "versions": {}}))
em = np.array([res[k]["em"] for k in names])
se = np.array([res[k]["em_stderr"] for k in names])
expected = np.sqrt(np.sum(se**2)) / len(se)
mc = (rng.binomial(N, p, size=(20000, 57)) / N).mean(axis=1).std()
for row in ("mmlu:_average|0", "all"):
print(f"{row:15s} em = {res[row]['em']:.5f} (mean of subtask em {em.mean():.5f}) | em_stderr: fixed "
f"{res[row]['em_stderr']:.5f} | expected {expected:.5f} | Monte Carlo SD {mc:.5f}")
assert np.isclose(res[row]["em"], em.mean()) # point estimate unchanged
assert np.isclose(res[row]["em_stderr"], expected) # stderr = sqrt(sum se_i^2) / G
assert abs(res[row]["em_stderr"] / mc - 1) < 0.05 # and matches the true sampling SD
print("fix OK")
Output:
| Task |Version|Metric|Value | |Stderr|
|---------------|-------|------|-----:|---|-----:|
|all | |em |0.6077|± |0.0047|
|mmlu:_average\|0| |em |0.6077|± |0.0047|
mmlu:_average|0 em = 0.60774 (mean of subtask em 0.60774) | em_stderr: fixed 0.00475 | expected 0.00475 | Monte Carlo SD 0.00473
all em = 0.60774 (mean of subtask em 0.60774) | em_stderr: fixed 0.00475 | expected 0.00475 | Monte Carlo SD 0.00473
fix OK
Version info
- lighteval 0.13.0 (PyPI release), numpy 2.5.4, torch 2.15.0+cpu
- Python 3.13.13, Linux x86_64, CPU only
I would be happy to open a PR with this fix.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 612
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 3
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from huggingface/lighteval
-
[BUG] arc_agi_2: a correct grid in the format the prompt asks for scores 0 (exact_match gold is a one-line list)Possibly taken @BahaaMattar claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
huggingface/lighteval#1510 ·
Maintainers usually reply within 1 day
-
[BUG] vllm_main_tests.yaml: continue-on-error hides a canary 23 scheduled runs deep in failurePossibly taken @voropaevv claimed this 9 days ago. Open
Difficulty 2/5 Under an hour Newbie friendliness 76/100
huggingface/lighteval#1486 ·
Maintainers usually reply within 1 day
-
[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literalsPossibly taken @inchang-ing claimed this 15 days ago. Openbug
Difficulty 1/5 1-3 hours Newbie friendliness 74/100
huggingface/lighteval#1471 ·
Maintainers usually reply within 1 day
-
[BUG] Reasoning tag removal fails when <think> is part of the input promptPossibly taken @selimfedakar claimed this 21 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/lighteval#1453 · 1 comment ·
Maintainers usually reply within 1 day
-
[BUG] TruthfulQA MC2 drops true answers after a gap in gold indexesPossibly taken @salma2vec claimed this 29 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
huggingface/lighteval#1442 ·
Maintainers usually reply within 1 day
All issues in huggingface/lighteval
Similar issues
-
integration: media_player
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
home-assistant/core#193344 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 82/100
daniel-ospina/tortoise#13607 ·
Maintainers usually reply within 1 day
-
documentation priority: medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day
-
needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day