Errata v4: Eval.ts score --check passes on a small gold set and fails on a larger one; Eval.ts sample cannot draw before a first calibration

Aperta
#2,752 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
bun, typescript

Direzione di ricerca

Start with LIFEOS/ERRATA/Eval.ts:127 and inspect how sample chooses failure strata and how score --check evaluates the gold set. Reproduce the two cases with the commands and dataset sizes in the issue. Done when fresh-install sampling can produce the requested draw and the check output makes the small-set failure count clear without misreporting the larger-set result.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Version

LifeOS 8.761.0 · attestation: not run

What is broken

Two things in Errata v4's eval tooling, both seen on one 8.761.0 install with a corpus of 1,231 typed exchanges.

  1. Sampler. Eval.ts sample --n 300 --failures 180 --per-session 3 stratifies on the composer at the registry cuts. With the shipped provisional cuts it answers only 12/180 failures. After one Train.ts --calibrate on a 150-row set the composer calls 371 failures and the draw works. The documented gold draw is unreachable on a fresh install.
  2. Gate. On a 150-row gold set (15 majority failures) check (gold) reads PASS. On a 280-row set (25 majority failures, three blind labelers, class agreement 92–95%) it reads FAIL: recall 88.9% and class given detection 87.5% clear their bars, but weighted detection accuracy is 89.3% against 94.5% for the regex read, difference [-10.1%, -0.7%].

Suggested fix: let sample stratify on the shipped picker, or take cuts as a flag, and print the number of majority failures next to PASS so a small set is not read as proof.

Where (file:line)

LIFEOS/ERRATA/Eval.ts:127

Repro on a clean tree
1. On an install whose `errata-triage` cuts are still the shipped provisional values, run `bun LIFEOS/ERRATA/Errata.ts exchanges` and `bun LIFEOS/ERRATA/Errata.ts v4 detect`.
2. Run `bun LIFEOS/ERRATA/Eval.ts sample --n 300 --failures 180 --per-session 3`. It stops with `sample: only N/180 failures` (N was 12 on a corpus of 1,231 exchanges).
3. Draw a small set instead, have three labelers label it, run `bun LIFEOS/ERRATA/Train.ts --calibrate`, then `bun LIFEOS/ERRATA/Eval.ts score --check --offline`: PASS on 150 rows with 15 majority failures.
4. Draw a larger set (`--n 280 --failures 160 --per-session 4 --force`), label it the same way, run the same check: FAIL, as in the captured probe.
Negative control

Probe: bun LIFEOS/ERRATA/Eval.ts score --check --offline 2>&1 | grep -E '^check|^gold:|regex frustrationRead|shipped . regex' · exit 0 · captured 2026-11-11T09:01:25.347Z

Red looks like: check (gold): FAIL

gold: 280 rows, 2 contested on class (left out of class scores); weights failures ×2.32, non-failures ×7.17; shipped picker: v4 combiner + class tier max, second look
regex frustrationRead                     94.5%      [91.7%, 96.7%]    90.4%     280
  detection  shipped − regex  [-10.1%, -0.7%]
check (gold): FAIL (v4 combiner + class tier max, second look must beat v3 on class, and v3 and the regex on detection, weighted; the bars gate the holdout, gold reads class given detection 87.5% (ISC-53, bar 80.0%), recall among majority failures 88.9% (ISC-56, bar 80.0%), end to end 77.9%)
Suggested fix

None proposed.

Environment
  • OS: macOS 27.2 (arm64)
  • Harness: claude-code
  • bun: 1.4.4 · git: 2.54.0
  • Doctor broken: none
Lingua principale
TypeScript
Stelle
19.9k
Fork
2.5k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di danielmiessler/LifeOS

Tutte le issue di danielmiessler/LifeOS

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.