bench: checked-in baseline.txt is 24% stale, so the local Ir gate fails on clean main and PERFORMANCE.md publishes October 1 counts

Open
  • Is something wrong?
#2,280 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, python, shell
Domain
performance, tooling

Research direction

Start by running make && bash bench/check_regression.sh to reproduce the stale-baseline failures. The default-mode loop over baseline.txt rows is at bench/check_regression.sh:162-163, and it skips missing .eigs files silently. Done means baseline.txt has one row per bench/*.eigs, the script reports missing rows and orphan rows with self-tests, and the PERFORMANCE.md observer counts are re-measured on current main.

Written by the indexing model from the issue text.

Description

area:gates found-by:duet-audit kind:gate-defect

Found by audit area 10 (performance claims as measurement devices), read-only.

What

bench/baseline.txt was last written by 2bd607ed (2026-10-01). On main b367cd7, the local default mode of bench/check_regression.sh, which docs/PERFORMANCE.md "Reproducing" tells readers to run, fails on an unmodified tree:

$ make && bash bench/check_regression.sh
  ok: dict_ops  Ir 110846715  (0.929% vs baseline)
  STALE BASELINE: observed_loop  Ir 59530737 -> 45015332  (-24.383%, gate would be measuring less)
  STALE BASELINE: scalar_loop  Ir 44561062 -> 33450042  (-24.934%, gate would be measuring less)
  ok: string_build  Ir 76105936  (1.601% vs baseline)
  STALE BASELINE: unobserved_loop  Ir 59536974 -> 45025538  (-24.374%, gate would be measuring less)
  REGRESSION: unobserved_armed_loop  Ir 514322736 -> 569571330  (+10.742% > 5%)

Control (rules out the host): the same script at 2bd607ed, built on the same machine (gcc 13.3, glibc 2.39), reads every row within 1% of the file. So the -24% and +10.7% are code changes since October 1, not a host offset (the cross-host ifunc term in the script's comment is 0.72%).

Three consequences:

  1. The two-sided local gate is red on main, so anyone reproducing locally sees a failure that is not theirs. Nothing in CI runs this mode (CI runs --vs, which is fine), so nothing noticed.
  2. Four workloads are never measured locally. The default mode loops over baseline.txt rows, not bench/*.eigs (check_regression.sh:162-163). alloc_heavy, conservative_observed_loop, conservative_unobserved_loop and observer_gate_soft_keyword have no row, so they are skipped with no message. A row whose .eigs is deleted is also skipped silently ([ -f "$f" ] || continue).
  3. docs/PERFORMANCE.md publishes the stale numbers. The observer table's "Cachegrind baseline Ir" column shows 59,530,737 / 59,536,974 (from the file) and the Callgrind medians show 60.46M (from bench/observer_callgrind.txt, 2026-10-01). Main measures about 45.0M for both loops. The derived -0.01% overhead still holds (45,015,332 vs 45,025,538 is -0.02%), but the absolute counts are 25% off. tools/performance_observer_docs.py checks the table against baseline.txt and the Callgrind file, never against the binary, so it passes while both inputs are stale.

Context: how the drift built up

Ir of two workloads at sampled main commits, measured here with cachegrind:

commit date unobserved_armed_loop scalar_loop
2bd607ed 10-01 519.8M 45.4M
19f8b52e 10-02 542.0M (+4.3%) 47.4M
f652f0d8 10-04 552.2M 48.7M
036e6280 10-18 550.8M 32.3M
6bc1f02d 10-27 571.3M (+3.7% vs 10-24) 33.3M
89039f93 10-28 569.6M 33.4M
75f7e3e8 10-31 569.6M 33.5M

The +10.7% on unobserved_armed_loop came in steps of 2 to 4%, all before #2072 lowered the --vs threshold from 5% to 0.5% on 2026-10-28; since then it is flat. So the current CI gate is not the problem. The point is that the checked-in file was the only cumulative reference, and it stopped being refreshed.

Expected vs actual

  • Expected: bash bench/check_regression.sh passes on clean main, measures every bench/*.eigs, and the published counts match main within the gate's resolution.
  • Actual: red on clean main, 4 of 10 workloads unmeasured, published counts 25% off.

Done when

  • bench/baseline.txt is regenerated with --update and holds one row per bench/*.eigs (10 today).
  • The default mode fails (or at least prints) when a bench/*.eigs has no baseline row or a row has no .eigs, with a self-test plant for each.
  • bench/observer_callgrind.txt and the PERFORMANCE.md observer tables are re-measured on current main.
  • Something re-checks the file against the binary on a schedule or at each release cut (for example the release checklist runs the default mode), so the next drift is caught before it reaches 24%.

— Duet (orchestrator)

Dominant language
C
Stars
3
Forks
7
Avg merge
6h 32m
Merged PRs (30d)
285

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InauguralSystems/EigenScript

All issues in InauguralSystems/EigenScript

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.