Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

PERFORMANCE.md states a 46% observer overhead that measures 0%, and the gate that should have caught it is one-sided by construction

Open
  • Is something wrong?
#1,206 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
65/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
c, shell
Domain
build-system, documentation, performance, testing-qa

Research direction

Start by reading docs/PERFORMANCE.md:138-140, bench/check_regression.sh, and bench/baseline.txt, then rerun the observer benchmark with n=5 Ir measurements. Check the observer-gate history referenced by #915 and PR #1034. Done means the current cost is established, improvements cannot silently pass the gate, and the documented figures are mechanically checked or generated from the baseline.

Written by the indexing model from the issue text.

Description

area:docs area:gates found-by:critic kind:gate-defect

Found by re-running a number the docs state in the present tense.

docs/PERFORMANCE.md:139 documents observer overhead as "~46% more (Ir
≈ 93.2M vs 63.8M)" and :138 as ~28% slower wall-clock.

Measured

Reported by Jon, then reproduced independently here (callgrind, this dev
box, current main build):

documented Jon reproduced here
observed_loop Ir 93,231,983 59,567,345 59,519,029 (−36.2%)
unobserved_loop Ir 63,843,716 59,573,261 59,525,213 (−6.8%)
observed − unobserved +46% −5,916 −6,184 (−0.01%)
wall 37 vs 29 ms 6 vs 6 ms (median of 15)

The observed loop now runs fewer instructions than the unobserved one.
That is noise around zero — not a win in the other direction — but the
documented cost of observation has gone from 46% to nothing, and the docs
still assert it in the present tense with a deterministic Ir figure
attached, which is the form most likely to be believed.

The two documented figures are byte-identical to bench/baseline.txt
(observed_loop 93231983, unobserved_loop 63843716), so this is one
staleness event surfacing in two places — the same shape as the
bench/baseline.txt staleness found this morning.

The gate cannot catch this, and that is the more useful half

bench/check_regression.sh compares one direction only:

inc=$(( (cur - base) * 100 / base ))
if [ "$inc" -gt "$THRESHOLD_PCT" ]; then   # THRESHOLD_PCT=5

A drop of any size prints ok. observed_loop currently reads
-36% vs baseline and the gate is green.

So a one-sided gate cannot distinguish a real 36% improvement from a
36%-wrong baseline — both render identically, as good news. That is why
one unrecorded regeneration event desynced three artifacts at once and
nothing spoke: the baseline, the gate's effective threshold, and the
published docs.

docs/PERFORMANCE.md:140 compounds it by claiming the win is "now
regression-gated in both directions". It is gated in one.

What to do (measurement pass, not a quick edit)

  1. Establish what the observer costs now, properly: n=5 vs n=5, and
    Ir rather than wall (Ir here was stable; the wall figures are 6 ms
    and unusable at this size). If it is genuinely ~0, find out when it
    went to zero — the #915 observer gate (PR #1034) is the obvious
    candidate and would mean unobserved: no longer buys anything on this
    workload, which is a real documentation change, not a number swap.
  2. Make the gate two-sided. An improvement beyond the threshold should
    require the baseline to be re-pinned deliberately, exactly as a
    regression does. A silent improvement is the signature of a stale
    baseline far more often than of a win.
  3. Make line 139 derived. check_regression.sh --update regenerates
    the baseline; nothing regenerates the docs, so the copy in
    PERFORMANCE.md will rot again on the next legitimate update. Either
    generate that table from bench/baseline.txt, or enrol both figures in
    tools/docs_claims_check.sh so the doc fails when it disagrees with
    the file it was copied from.

Note on what is NOT wrong here

The section immediately above (lines 100–120) is careful work: thresholds
placed from measured spreads, false-red rates counted across 14 and 5
runs, a --selftest that proves the check can fail. It was built to be
re-run. Lines 121–139 are a table from "a 2020-era x86-64 Linux dev box"
that can only be re-run by someone who thinks to.

That is the distinction worth keeping: prose lessons age well, measured
claims rot silently — and they rot in whatever copies were made of them.
Same point as the stale CLAUDE.md passage in ouroboros, now with a second
instance and a mechanical cause.

Dominant language
C
Stars
3
Forks
7
Avg merge
3h 56m
Merged PRs (30d)
102

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InauguralSystems/EigenScript

All issues in InauguralSystems/EigenScript

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.