Errata v4: Eval.ts score --check passes on a small gold set and fails on a larger one; Eval.ts sample cannot draw before a first calibration

Open
#2,752 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
bun, typescript

Research direction

Start with LIFEOS/ERRATA/Eval.ts:127 and inspect how sample chooses failure strata and how score --check evaluates the gold set. Reproduce the two cases with the commands and dataset sizes in the issue. Done when fresh-install sampling can produce the requested draw and the check output makes the small-set failure count clear without misreporting the larger-set result.

Written by the indexing model from the issue text.

Description

Version

LifeOS 8.761.0 · attestation: not run

What is broken

Two things in Errata v4's eval tooling, both seen on one 8.761.0 install with a corpus of 1,231 typed exchanges.

  1. Sampler. Eval.ts sample --n 300 --failures 180 --per-session 3 stratifies on the composer at the registry cuts. With the shipped provisional cuts it answers only 12/180 failures. After one Train.ts --calibrate on a 150-row set the composer calls 371 failures and the draw works. The documented gold draw is unreachable on a fresh install.
  2. Gate. On a 150-row gold set (15 majority failures) check (gold) reads PASS. On a 280-row set (25 majority failures, three blind labelers, class agreement 92–95%) it reads FAIL: recall 88.9% and class given detection 87.5% clear their bars, but weighted detection accuracy is 89.3% against 94.5% for the regex read, difference [-10.1%, -0.7%].

Suggested fix: let sample stratify on the shipped picker, or take cuts as a flag, and print the number of majority failures next to PASS so a small set is not read as proof.

Where (file:line)

LIFEOS/ERRATA/Eval.ts:127

Repro on a clean tree
1. On an install whose `errata-triage` cuts are still the shipped provisional values, run `bun LIFEOS/ERRATA/Errata.ts exchanges` and `bun LIFEOS/ERRATA/Errata.ts v4 detect`.
2. Run `bun LIFEOS/ERRATA/Eval.ts sample --n 300 --failures 180 --per-session 3`. It stops with `sample: only N/180 failures` (N was 12 on a corpus of 1,231 exchanges).
3. Draw a small set instead, have three labelers label it, run `bun LIFEOS/ERRATA/Train.ts --calibrate`, then `bun LIFEOS/ERRATA/Eval.ts score --check --offline`: PASS on 150 rows with 15 majority failures.
4. Draw a larger set (`--n 280 --failures 160 --per-session 4 --force`), label it the same way, run the same check: FAIL, as in the captured probe.
Negative control

Probe: bun LIFEOS/ERRATA/Eval.ts score --check --offline 2>&1 | grep -E '^check|^gold:|regex frustrationRead|shipped . regex' · exit 0 · captured 2026-11-11T09:01:25.347Z

Red looks like: check (gold): FAIL

gold: 280 rows, 2 contested on class (left out of class scores); weights failures ×2.32, non-failures ×7.17; shipped picker: v4 combiner + class tier max, second look
regex frustrationRead                     94.5%      [91.7%, 96.7%]    90.4%     280
  detection  shipped − regex  [-10.1%, -0.7%]
check (gold): FAIL (v4 combiner + class tier max, second look must beat v3 on class, and v3 and the regex on detection, weighted; the bars gate the holdout, gold reads class given detection 87.5% (ISC-53, bar 80.0%), recall among majority failures 88.9% (ISC-56, bar 80.0%), end to end 77.9%)
Suggested fix

None proposed.

Environment
  • OS: macOS 27.2 (arm64)
  • Harness: claude-code
  • bun: 1.4.4 · git: 2.54.0
  • Doctor broken: none
Dominant language
TypeScript
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from danielmiessler/LifeOS

All issues in danielmiessler/LifeOS

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.