bench: checked-in baseline.txt is 24% stale, so the local Ir gate fails on clean main and PERFORMANCE.md publishes October 1 counts

未关闭
  • 哪里不对？
#2,280 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
c, python, shell
领域
performance, tooling

调研方向

Start by running make && bash bench/check_regression.sh to reproduce the stale-baseline failures. The default-mode loop over baseline.txt rows is at bench/check_regression.sh:162-163, and it skips missing .eigs files silently. Done means baseline.txt has one row per bench/*.eigs, the script reports missing rows and orphan rows with self-tests, and the PERFORMANCE.md observer counts are re-measured on current main.

由索引模型根据 Issue 内容生成。

描述

area:gates found-by:duet-audit kind:gate-defect

Found by audit area 10 (performance claims as measurement devices), read-only.

What

bench/baseline.txt was last written by 2bd607ed (2026-10-01). On main b367cd7, the local default mode of bench/check_regression.sh, which docs/PERFORMANCE.md "Reproducing" tells readers to run, fails on an unmodified tree:

$ make && bash bench/check_regression.sh
  ok: dict_ops  Ir 110846715  (0.929% vs baseline)
  STALE BASELINE: observed_loop  Ir 59530737 -> 45015332  (-24.383%, gate would be measuring less)
  STALE BASELINE: scalar_loop  Ir 44561062 -> 33450042  (-24.934%, gate would be measuring less)
  ok: string_build  Ir 76105936  (1.601% vs baseline)
  STALE BASELINE: unobserved_loop  Ir 59536974 -> 45025538  (-24.374%, gate would be measuring less)
  REGRESSION: unobserved_armed_loop  Ir 514322736 -> 569571330  (+10.742% > 5%)

Control (rules out the host): the same script at 2bd607ed, built on the same machine (gcc 13.3, glibc 2.39), reads every row within 1% of the file. So the -24% and +10.7% are code changes since October 1, not a host offset (the cross-host ifunc term in the script's comment is 0.72%).

Three consequences:

  1. The two-sided local gate is red on main, so anyone reproducing locally sees a failure that is not theirs. Nothing in CI runs this mode (CI runs --vs, which is fine), so nothing noticed.
  2. Four workloads are never measured locally. The default mode loops over baseline.txt rows, not bench/*.eigs (check_regression.sh:162-163). alloc_heavy, conservative_observed_loop, conservative_unobserved_loop and observer_gate_soft_keyword have no row, so they are skipped with no message. A row whose .eigs is deleted is also skipped silently ([ -f "$f" ] || continue).
  3. docs/PERFORMANCE.md publishes the stale numbers. The observer table's "Cachegrind baseline Ir" column shows 59,530,737 / 59,536,974 (from the file) and the Callgrind medians show 60.46M (from bench/observer_callgrind.txt, 2026-10-01). Main measures about 45.0M for both loops. The derived -0.01% overhead still holds (45,015,332 vs 45,025,538 is -0.02%), but the absolute counts are 25% off. tools/performance_observer_docs.py checks the table against baseline.txt and the Callgrind file, never against the binary, so it passes while both inputs are stale.

Context: how the drift built up

Ir of two workloads at sampled main commits, measured here with cachegrind:

commit date unobserved_armed_loop scalar_loop
2bd607ed 10-01 519.8M 45.4M
19f8b52e 10-02 542.0M (+4.3%) 47.4M
f652f0d8 10-04 552.2M 48.7M
036e6280 10-18 550.8M 32.3M
6bc1f02d 10-27 571.3M (+3.7% vs 10-24) 33.3M
89039f93 10-28 569.6M 33.4M
75f7e3e8 10-31 569.6M 33.5M

The +10.7% on unobserved_armed_loop came in steps of 2 to 4%, all before #2072 lowered the --vs threshold from 5% to 0.5% on 2026-10-28; since then it is flat. So the current CI gate is not the problem. The point is that the checked-in file was the only cumulative reference, and it stopped being refreshed.

Expected vs actual

  • Expected: bash bench/check_regression.sh passes on clean main, measures every bench/*.eigs, and the published counts match main within the gate's resolution.
  • Actual: red on clean main, 4 of 10 workloads unmeasured, published counts 25% off.

Done when

  • bench/baseline.txt is regenerated with --update and holds one row per bench/*.eigs (10 today).
  • The default mode fails (or at least prints) when a bench/*.eigs has no baseline row or a row has no .eigs, with a self-test plant for each.
  • bench/observer_callgrind.txt and the PERFORMANCE.md observer tables are re-measured on current main.
  • Something re-checks the file against the binary on a schedule or at each release cut (for example the release checklist runs the default mode), so the next drift is caught before it reaches 24%.

— Duet (orchestrator)

主要语言
C
星标
3
派生
7
平均合并
6 小时 32 分钟
30 天内合并 PR
285

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

InauguralSystems/EigenScript 的其他 Issue

查看 InauguralSystems/EigenScript 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。