Regression - BUGFIX - Fix two silent failure modes: PATH julia resolution and single-sample result caching

Open
  • Is something wrong?
#418 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
julia, sqlite
Domain
testing

Research direction

Read regression-harness/src/env.jl:66 and regression-harness/src/runner.jl:278, :480, and :597 to trace Julia resolution and subprocess launches; then inspect regression-harness/src/database.jl:24 and regress.jl:196 for cache and repeated-ref behavior. The issue offers several remedies rather than selecting one, so first decide the intended behavior for both failure modes. Done means probes cannot hang silently and reports or sampling make cached results distinguishable from fresh measurements.

Written by the indexing model from the issue text.

Description

The regression harness has two independent ways of misleading a reader silently. Both were hit on 2026-08-21, in two separate sessions, and both produced output that would have gone into a pull request body as evidence.

Provenance: the PATH hang was found by the layer-overlap (#298) investigation; the cache-propagation mechanism was diagnosed by the Δ' reference-length (#403) investigation (shared-cached-row observation and the UNIQUE schema read) and confirmed by the layer-overlap investigation's forced experiment. Related but distinct from #338 (Manifest drift across refs), which is a third way the same comparison can mislead.

1. A bare julia from PATH can hang forever

The harness resolves julia from PATH in four places: the version probe at regression-harness/src/env.jl:66, and the three GPEC subprocess launches at regression-harness/src/runner.jl:278, :480 and :597.

Where julia on PATH is the juliaup shim (~/.juliaup/bin/julialauncher), the probe hangs indefinitely rather than returning. Two consecutive runs sat for ~18 min and ~4 min producing no output, indistinguishable from a long precompile. The process tree is the only tell — parent at 0% CPU with a stuck child:

3997  julia --project=regression-harness regression-harness/regress.jl ...   0.0% CPU
 4024  /bin/sh /Users/.../.juliaup/bin/julialauncher --version               0.0% CPU

subprocess_julia_version() already wraps the probe in try/catch returning "unknown", so graceful degradation was intended. A hang is not an exception, so that guard never fires.

Remedies:

  • Resolve the interpreter from the running process rather than PATH — Base.julia_cmd() or joinpath(Sys.BINDIR, "julia"). The harness is already running under a working Julia, so the correct binary is known and need not be rediscovered. Fixes all four sites.
  • Put a timeout on the --version probe so it errors instead of wedging if ever spawned externally. "unknown" was the right answer and should be reachable.

Workaround: export PATH="$HOME/.julia/juliaup/julia-<version>/bin:$PATH" before invoking. Passing an absolute path to regress.jl is not sufficient — the hang is in the child it spawns.

2. One cached sample per commit becomes that commit's canonical value

runs is declared UNIQUE(commit_hash, case_name) (regression-harness/src/database.jl:24), so exactly one row exists per (SHA, case). Whichever sample is taken first becomes that SHA's value until someone passes --force, and every later comparison against that SHA inherits it.

For a deterministic case that is a pure win. For one with any run-to-run variation it silently propagates a fixed offset into unrelated comparisons. Measured on diiid_slayer_n1, same case, same two refs, cached versus both sides forced:

quantity vs. cached row both sides forced ratio
Q_root 5.6e-06 5.0e-10 11,000x
omega_Hz 1.9e-01 1.1e-05 17,000x
gamma_Hz 7.6e-02 1.5e-05 5,000x

All 16 layer-input quantities were bit-identical in both runs, so no source effect was involved. The residual after forcing is four orders below the cached figure.

The cross-session evidence is the clearest part: two different branches, compared by two different people against the same cached 06be78884 row, both reported omega_Hz = 1.9e-01. An identical figure from two unrelated source deltas is the cache, not physics. Both were on course to be quoted in PR bodies as real movement.

Remedies, in increasing order of ambition:

  • Print provenance per side in the report — the runs table already carries a timestamp, so marking each column cached or fresh makes a poisoned comparison visible rather than invisible.
  • Imply --force for the ref that equals the working tree's HEAD when producing a stamp for a pull request, since that is the side a reviewer is being asked to trust.
  • Consider allowing more than one sample per (SHA, case) so a repeat run measures the reproducibility floor instead of overwriting it. Today --refs X,X cannot measure it: the ref list is not deduplicated (regress.jl:196), but the UNIQUE key means both sides read the same surviving row and the report prints exactly 0.0.
The artifact selectively poisons the comparisons that matter most

Measured across four comparisons on the same case, the cached sample's jitter is only visible where the genuine source effect is at or below sample scale:

comparison genuine mechanism cache artifact visible?
K^(2mu) reference-length conversion (#403) 42% gamma no — reproduces to 2 digits under --force
Delta' shift from the shooting tolerance (#413) 0.02% gamma no — reproduces to 4 digits under --force
analytic-derivative swap only ~nil yes — omega 1.9e-01, collapses to 1.8e-03 forced
layer-width additions (no SLAYER path, #298) nil yes — omega 1.9e-01, collapses to 1.1e-05 forced

Large real effects punch straight through the artifact. Null and near-null results are dominated by it. That is the worst possible selectivity, because "did anything change?" is the question the harness exists to answer, and a null result is the answer most often taken at face value.

Why these belong together

Both fail silently in a tool whose entire purpose is producing a numerical claim a reviewer will trust. Neither produces an error, a warning, or a visible anomaly. The first costs time; the second costs correctness, and did so in two sessions on the same day without either noticing until the same number appeared twice.

Dominant language
Julia
Stars
11
Forks
7
Avg merge
4d 3h
Merged PRs (30d)
70

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenFUSIONToolkit/GPEC

All issues in OpenFUSIONToolkit/GPEC

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.