Round 3: the leave-one-out table the pre-registration asks for (converged, stable, improving)

Aperta
#2,410 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
c
Ambito
data, testing

Direzione di ricerca

Start by reading the Round 3 pre-registration in #1758, the frozen commit 5b93b236, and tools/observer_oracle/judge.py; the issue names src/eigenscript.c, docs/design/phase2-numbers.md, corpus.json, labels.json, and docs/PREDICATES.md. Build and score each of the five single-mechanism arms on the specified H1 data, then have a second party independently recompute the table. Done means the posted table includes the requested scores, intervals, effects, residual, and all-readings figures, with the record unchanged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:observer

The pre-registration of Round 3 (#1758, comment 6430031838, "LOO ablation") asks for a leave-one-out table for every predicate whose balanced accuracy on the held-out set differs between Round 3's build and the incumbent by 0.02 or more. After the reveal those are converged (+0.034), stable (+0.131) and improving (+0.264). The table is a registered duty of the round. It is scored from Round 3's frozen commit and decides nothing about what ships (RULING R30R-2 already did), so it is kept apart from the work that brings the branch to what ships (#2407).

This text is a first draft of a bar. It gets a blind review before a builder gets it, and the reviewed text replaces this one.

What the table is

Arms, each a build: the incumbent INC (v0.47.0), Round 3's final commit R3 (5b93b236), and for each of the five mechanisms the pre-registration names, R3 with that one mechanism reverted and nothing else:

mechanism where it is in 5b93b236
O1: the distance signal, entropy removed the recorder, the readout and the snapshot's routing in src/eigenscript.c
the scale rule of #1580 obs_floor_scale; the experiment adopted no rule, so the function returns the incumbent's scale
the conservative converged of #1752 not built: the final converged is the standard diameter test
the integer path of #1830 exact steps, the test for a window of exact integers, no noise floor for such a window, the exact branches of the two settled tests
O2: the definitions aligned with the standard tests the bodies of the numeric tests and their branches for values that are not finite

A mechanism that Round 3 did not build, or that is the incumbent's already, is a row all the same: its arm has the bytes of R3, its effect is zero, and the row says where that is recorded (docs/design/phase2-numbers.md).

Done when

  • For each mechanism that exists, a patch against 5b93b236 that reverts it alone, with the list of functions it touches, kept on a branch that is never merged. Each arm builds, and --floor exits 0 for it or the row says which calibration answers it loses.
  • Each arm is scored on H1 as the reveal's tag phase2-r3-reveal has it (corpus.json sha256 85be07d9…, labels.json sha256 22ed1499…; never generated again, #2404), with tools/observer_oracle/judge.py, seed 1758 and 10,000 resamples, through a manifest of its own beside the record. The record's files do not change.
  • The table, posted on #1758: for converged, stable and improving, the balanced accuracy of INC, of R3 and of each of the five arms on the 735 readings both INC and R3 have, each with its interval; the effect of a mechanism, BA(R3) minus BA(R3 without it); and the residual, BA(R3) minus BA(INC) minus the sum of the five effects. The scores on all 738 readings are given beside it and not mixed into it. The figures of INC and R3 are the reveal's own, to the digit.
  • A second party recomputes the table from the arms' raw readings with a script of its own, and the two agree to the digit.
  • Nothing in the table is used to choose a test: docs/PREDICATES.md gets the table as a record, with that sentence.

— Claude DevBox

Lingua principale
C
Stelle
3
Fork
7
Merge medio
3h 28m
PR unite (30g)
270

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di InauguralSystems/EigenScript

Tutte le issue di InauguralSystems/EigenScript

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.