[BUG] Stderr of `_average` and `all` rows is the mean of the subtask stderrs (about 7x too large for MMLU)

Open Beginner friendly
#1,466 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@Andreas-Hum is already working on this.

Since Nov 1, 2026.

  • #1468 by @Andreas-Hum — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
70/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python

Research direction

Start in MetricsLogger.aggregate in src/lighteval/logging/info_loggers.py, where the _average group rows and the all row sum every metric key, including *_stderr. Run the repro script in the issue to confirm the stderr equals the mean of the subtask stderrs. Done means _stderr keys combine as sqrt(sum of squares) / G while point estimates stay the same, which the issue's fix_check script asserts.

Written by the indexing model from the issue text.

Description

Describe the bug

For task families with several subtasks, MetricsLogger.aggregate builds a group row (e.g. mmlu:_average|5) and the all row by averaging every key of the subtask results, including the <metric>_stderr keys. The stderr shown for these rows is therefore the arithmetic mean of the subtask stderrs, not the standard error of the averaged score.

The averaged score is the unweighted mean of G subtask scores computed on disjoint question sets, so its standard error is sqrt(sum_i se_i^2) / G. The mean of the stderrs is larger by the factor sum_i se_i / sqrt(sum_i se_i^2), which lies between 1 and sqrt(G). For the 57 MMLU subjects it is about 7.3 (sqrt(57) = 7.55): in the repro below the results table shows 0.6077 ± 0.0345 (95% interval ±6.8 points) while the standard error of that average is 0.0047 (±0.9 points). Every multi-subtask family (mmlu:*, bigbench_hard:*, bigbench:*, ...) is affected in its _average row and in all.

To Reproduce

The script below feeds synthetic per-question scores for the 57 real MMLU task configs (with the MMLU test-set sizes) through MetricsLogger.log / MetricsLogger.aggregate and prints the rows with make_results_table. No model or dataset download is needed.

import numpy as np
from lighteval.logging.info_loggers import MetricsLogger
from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.tasks.mmlu import TASKS_TABLE
from lighteval.utils.utils import make_results_table

# MMLU test-set sizes of the 57 subjects (alphabetical order, 14042 questions)
N = np.array([100, 135, 152, 100, 265, 144, 100, 100, 100, 173, 102, 100, 235, 114, 145, 378, 126, 100, 310,
              203, 100, 165, 198, 193, 390, 270, 238, 151, 545, 216, 204, 237, 223, 131, 121, 108, 163, 112,
              103, 234, 100, 783, 346, 895, 306, 311, 324, 282, 1534, 272, 612, 110, 245, 201, 100, 166, 171])
tasks = {cfg.full_name: LightevalTask(cfg) for cfg in TASKS_TABLE}  # the 57 real MMLU tasks (no data download)
names = sorted(tasks)
assert len(names) == len(N) == 57 and N.sum() == 14042

rng = np.random.default_rng(0)
p = rng.uniform(0.35, 0.85, size=57)  # synthetic per-subject accuracy
logger = MetricsLogger()
for name, n, p_k in zip(names, N, p):
    for correct in rng.random(n) < p_k:  # synthetic per-question exact-match scores
        logger.log(name, {"em": float(correct)})
logger.aggregate(task_dict=tasks, bootstrap_iters=1000)
res = logger.metric_aggregated
print(make_results_table({"results": {k: res[k] for k in ("all", "mmlu:_average|0")}, "versions": {}}))

se = np.array([res[k]["em_stderr"] for k in names])
expected = np.sqrt(np.sum(se**2)) / len(se)  # SE of the mean of 57 independent subtask means
mc = (rng.binomial(N, p, size=(20000, 57)) / N).mean(axis=1).std()  # SD of the score over fresh question draws
for row in ("mmlu:_average|0", "all"):
    actual = res[row]["em_stderr"]
    print(f"{row:15s} em_stderr: actual {actual:.5f} | expected sqrt(sum se_i^2)/G = {expected:.5f} "
          f"(Monte Carlo SD {mc:.5f}) | actual/expected = {actual / expected:.2f}")
    assert np.isclose(actual, se.mean())  # actual = arithmetic mean of the 57 subtask stderrs
    assert actual / expected > 5  # inflated ~7x (the limit for equal subtask stderrs is sqrt(57) = 7.55)

Output:

|     Task      |Version|Metric|Value |   |Stderr|
|---------------|-------|------|-----:|---|-----:|
|all            |       |em    |0.6077|±  |0.0345|
|mmlu:_average\|0|       |em    |0.6077|±  |0.0345|

mmlu:_average|0 em_stderr: actual 0.03452 | expected sqrt(sum se_i^2)/G = 0.00475 (Monte Carlo SD 0.00473) | actual/expected = 7.27
all             em_stderr: actual 0.03452 | expected sqrt(sum se_i^2)/G = 0.00475 (Monte Carlo SD 0.00473) | actual/expected = 7.27

The reported em_stderr of mmlu:_average|0 and all (0.03452) is exactly the arithmetic mean of the 57 subtask stderrs.

Expected behavior

The stderr of an averaged row should be the standard error of that average, sqrt(sum_i se_i^2) / G, which is also what a Monte Carlo over fresh question draws gives:

|all            |       |em    |0.6077|±  |0.0047|
|mmlu:_average\|0|       |em    |0.6077|±  |0.0047|

lm-evaluation-harness, which metrics/utils/stderr.py is adapted from, uses this formula for unweighted group means (unweighted_mean_stderr, lm_eval/api/metrics.py#L643-L651 at v0.4.13).

Root cause

src/lighteval/logging/info_loggers.py#L374-L402 (v0.13.0, 7c1cd62):

  • group rows, L393-L396: metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks) is applied to every key, including <metric>_stderr;
  • all row, L385-L387 and L399-L400: the same sum-then-divide for every key.

The rows are then printed as Value ± Stderr by make_results_table. The code is unchanged on main (b97a653, same lines).

Proposed fix

For _stderr keys, add the squares and take the square root before dividing by the number of subtasks (independent errors add in quadrature). Point estimates are unchanged.

--- a/src/lighteval/logging/info_loggers.py
+++ b/src/lighteval/logging/info_loggers.py
@@ -383,7 +383,9 @@
                 task, fewshot = k.split("|")
                 grouped_tasks[f"{task.split(':')[0]}:_average|{fewshot}"].append(k)
             for metric, value in metrics.items():
-                suite_average[metric] = suite_average.get(metric, 0) + value
+                suite_average[metric] = suite_average.get(metric, 0) + (
+                    value**2 if metric.endswith("_stderr") else value
+                )
                 suite_nb[metric] = suite_nb.get(metric, 0) + 1
 
         # Compute average for sub groups
@@ -391,13 +393,18 @@
             if len(list_of_subtasks) > 1:
                 metrics = list(self.metric_aggregated[list_of_subtasks[0]].keys())
                 self.metric_aggregated[average_task] = {
-                    metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks)
+                    metric: (
+                        sum(self.metric_aggregated[k][metric] ** 2 for k in list_of_subtasks) ** 0.5
+                        if metric.endswith("_stderr")
+                        else sum(self.metric_aggregated[k][metric] for k in list_of_subtasks)
+                    )
+                    / len(list_of_subtasks)
                     for metric in metrics
                 }
 
         # Compute average for all
         for metric, value in suite_average.items():
-            suite_average[metric] = value / suite_nb[metric]
+            suite_average[metric] = (value**0.5 if metric.endswith("_stderr") else value) / suite_nb[metric]
 
         self.metric_aggregated["all"] = suite_average
 

For all this assumes the tasks in the run are evaluated on different questions, which holds unless the same benchmark is included twice (e.g. at two few-shot settings).

Verifying the fix

fix_check.py applies the diff above to the installed MetricsLogger.aggregate and re-runs the same synthetic data:

fix_check.py
import inspect
import textwrap

import numpy as np
import lighteval.logging.info_loggers as info_loggers
from lighteval.logging.info_loggers import MetricsLogger
from lighteval.tasks.lighteval_task import LightevalTask
from lighteval.tasks.tasks.mmlu import TASKS_TABLE
from lighteval.utils.utils import make_results_table

# Apply the proposed diff to the installed MetricsLogger.aggregate (stderrs of independent subtasks add in quadrature)
FIX = [
    ("suite_average[metric] = suite_average.get(metric, 0) + value",
     'suite_average[metric] = suite_average.get(metric, 0) + (\n'
     '    value**2 if metric.endswith("_stderr") else value\n)'),
    ("metric: sum(self.metric_aggregated[k][metric] for k in list_of_subtasks) / len(list_of_subtasks)",
     "metric: (\n"
     "    sum(self.metric_aggregated[k][metric] ** 2 for k in list_of_subtasks) ** 0.5\n"
     '    if metric.endswith("_stderr")\n'
     "    else sum(self.metric_aggregated[k][metric] for k in list_of_subtasks)\n"
     ")\n/ len(list_of_subtasks)"),
    ("suite_average[metric] = value / suite_nb[metric]",
     'suite_average[metric] = (value**0.5 if metric.endswith("_stderr") else value) / suite_nb[metric]'),
]
src = textwrap.dedent(inspect.getsource(MetricsLogger.aggregate))
for old, new in FIX:
    assert src.count(old) == 1, old
    src = src.replace(old, new)
namespace = {}
exec(src, vars(info_loggers), namespace)
MetricsLogger.aggregate = namespace["aggregate"]

# Same synthetic data as repro.py
N = np.array([100, 135, 152, 100, 265, 144, 100, 100, 100, 173, 102, 100, 235, 114, 145, 378, 126, 100, 310,
              203, 100, 165, 198, 193, 390, 270, 238, 151, 545, 216, 204, 237, 223, 131, 121, 108, 163, 112,
              103, 234, 100, 783, 346, 895, 306, 311, 324, 282, 1534, 272, 612, 110, 245, 201, 100, 166, 171])
tasks = {cfg.full_name: LightevalTask(cfg) for cfg in TASKS_TABLE}
names = sorted(tasks)
rng = np.random.default_rng(0)
p = rng.uniform(0.35, 0.85, size=57)
logger = MetricsLogger()
for name, n, p_k in zip(names, N, p):
    for correct in rng.random(n) < p_k:
        logger.log(name, {"em": float(correct)})
logger.aggregate(task_dict=tasks, bootstrap_iters=1000)
res = logger.metric_aggregated
print(make_results_table({"results": {k: res[k] for k in ("all", "mmlu:_average|0")}, "versions": {}}))

em = np.array([res[k]["em"] for k in names])
se = np.array([res[k]["em_stderr"] for k in names])
expected = np.sqrt(np.sum(se**2)) / len(se)
mc = (rng.binomial(N, p, size=(20000, 57)) / N).mean(axis=1).std()
for row in ("mmlu:_average|0", "all"):
    print(f"{row:15s} em = {res[row]['em']:.5f} (mean of subtask em {em.mean():.5f}) | em_stderr: fixed "
          f"{res[row]['em_stderr']:.5f} | expected {expected:.5f} | Monte Carlo SD {mc:.5f}")
    assert np.isclose(res[row]["em"], em.mean())  # point estimate unchanged
    assert np.isclose(res[row]["em_stderr"], expected)  # stderr = sqrt(sum se_i^2) / G
    assert abs(res[row]["em_stderr"] / mc - 1) < 0.05  # and matches the true sampling SD
print("fix OK")

Output:

|     Task      |Version|Metric|Value |   |Stderr|
|---------------|-------|------|-----:|---|-----:|
|all            |       |em    |0.6077|±  |0.0047|
|mmlu:_average\|0|       |em    |0.6077|±  |0.0047|

mmlu:_average|0 em = 0.60774 (mean of subtask em 0.60774) | em_stderr: fixed 0.00475 | expected 0.00475 | Monte Carlo SD 0.00473
all             em = 0.60774 (mean of subtask em 0.60774) | em_stderr: fixed 0.00475 | expected 0.00475 | Monte Carlo SD 0.00473
fix OK

Version info

  • lighteval 0.13.0 (PyPI release), numpy 2.5.4, torch 2.15.0+cpu
  • Python 3.13.13, Linux x86_64, CPU only

I would be happy to open a PR with this fix.

Dominant language
Python
Stars
2.6k
Forks
612
Avg merge
2d 3h
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from huggingface/lighteval

All issues in huggingface/lighteval

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.