bench: checked-in baseline.txt is 24% stale, so the local Ir gate fails on clean main and PERFORMANCE.md publishes October 1 counts

Aberta
  • Algo errado?
#2,280 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
3/5
Tempo estimado
1-2 dias
Facilidade para iniciantes
35/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
c, python, shell
Domínio
performance, tooling

Direção de pesquisa

Start by running make && bash bench/check_regression.sh to reproduce the stale-baseline failures. The default-mode loop over baseline.txt rows is at bench/check_regression.sh:162-163, and it skips missing .eigs files silently. Done means baseline.txt has one row per bench/*.eigs, the script reports missing rows and orphan rows with self-tests, and the PERFORMANCE.md observer counts are re-measured on current main.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

area:gates found-by:duet-audit kind:gate-defect

Found by audit area 10 (performance claims as measurement devices), read-only.

What

bench/baseline.txt was last written by 2bd607ed (2026-10-01). On main b367cd7, the local default mode of bench/check_regression.sh, which docs/PERFORMANCE.md "Reproducing" tells readers to run, fails on an unmodified tree:

$ make && bash bench/check_regression.sh
  ok: dict_ops  Ir 110846715  (0.929% vs baseline)
  STALE BASELINE: observed_loop  Ir 59530737 -> 45015332  (-24.383%, gate would be measuring less)
  STALE BASELINE: scalar_loop  Ir 44561062 -> 33450042  (-24.934%, gate would be measuring less)
  ok: string_build  Ir 76105936  (1.601% vs baseline)
  STALE BASELINE: unobserved_loop  Ir 59536974 -> 45025538  (-24.374%, gate would be measuring less)
  REGRESSION: unobserved_armed_loop  Ir 514322736 -> 569571330  (+10.742% > 5%)

Control (rules out the host): the same script at 2bd607ed, built on the same machine (gcc 13.3, glibc 2.39), reads every row within 1% of the file. So the -24% and +10.7% are code changes since October 1, not a host offset (the cross-host ifunc term in the script's comment is 0.72%).

Three consequences:

  1. The two-sided local gate is red on main, so anyone reproducing locally sees a failure that is not theirs. Nothing in CI runs this mode (CI runs --vs, which is fine), so nothing noticed.
  2. Four workloads are never measured locally. The default mode loops over baseline.txt rows, not bench/*.eigs (check_regression.sh:162-163). alloc_heavy, conservative_observed_loop, conservative_unobserved_loop and observer_gate_soft_keyword have no row, so they are skipped with no message. A row whose .eigs is deleted is also skipped silently ([ -f "$f" ] || continue).
  3. docs/PERFORMANCE.md publishes the stale numbers. The observer table's "Cachegrind baseline Ir" column shows 59,530,737 / 59,536,974 (from the file) and the Callgrind medians show 60.46M (from bench/observer_callgrind.txt, 2026-10-01). Main measures about 45.0M for both loops. The derived -0.01% overhead still holds (45,015,332 vs 45,025,538 is -0.02%), but the absolute counts are 25% off. tools/performance_observer_docs.py checks the table against baseline.txt and the Callgrind file, never against the binary, so it passes while both inputs are stale.

Context: how the drift built up

Ir of two workloads at sampled main commits, measured here with cachegrind:

commit date unobserved_armed_loop scalar_loop
2bd607ed 10-01 519.8M 45.4M
19f8b52e 10-02 542.0M (+4.3%) 47.4M
f652f0d8 10-04 552.2M 48.7M
036e6280 10-18 550.8M 32.3M
6bc1f02d 10-27 571.3M (+3.7% vs 10-24) 33.3M
89039f93 10-28 569.6M 33.4M
75f7e3e8 10-31 569.6M 33.5M

The +10.7% on unobserved_armed_loop came in steps of 2 to 4%, all before #2072 lowered the --vs threshold from 5% to 0.5% on 2026-10-28; since then it is flat. So the current CI gate is not the problem. The point is that the checked-in file was the only cumulative reference, and it stopped being refreshed.

Expected vs actual

  • Expected: bash bench/check_regression.sh passes on clean main, measures every bench/*.eigs, and the published counts match main within the gate's resolution.
  • Actual: red on clean main, 4 of 10 workloads unmeasured, published counts 25% off.

Done when

  • bench/baseline.txt is regenerated with --update and holds one row per bench/*.eigs (10 today).
  • The default mode fails (or at least prints) when a bench/*.eigs has no baseline row or a row has no .eigs, with a self-test plant for each.
  • bench/observer_callgrind.txt and the PERFORMANCE.md observer tables are re-measured on current main.
  • Something re-checks the file against the binary on a schedule or at each release cut (for example the release checklist runs the default mode), so the next drift is caught before it reaches 24%.

— Duet (orchestrator)

Linguagem predominante
C
Estrelas
3
Forks
7
Merge médio
6h 32min
PRs com merge (30d)
285

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de InauguralSystems/EigenScript

Todas as issues de InauguralSystems/EigenScript

Issues semelhantes

Mais issues de C

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.