investigation(qa): pre-register #1068's section leave-one-out for qa on fixture #34, with the cost estimated before any run — written in the research repo, hash and estimate posted on #1068

Aperta
  • Qualcosa non va?
#1,653 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
18/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github, typescript
Ambito
developer-experience

Direzione di ricerca

Start with the blocked issue #1645 and the research repository’s rules for experiments and decisions; this issue requires work in the separate private admarble/smpl-sequant repository. The requested files are experiments/E8-qa-leave-one-out.md, decisions/log.md, and journal.md, plus a comment on #1068. Done means a committed, owner-approved preregistration with its cost estimate and a comment carrying its hash, arms, run count, and estimate without a fixture identifier; the issue says a separate owner decision is required before any run.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

graph-2026-11-simplify

Epic: graph-2026-11-simplify Tier: judgment Execution: runner Blocked by: #1645

  • Blocked by: #1645

Problem

D-4: pre-register #1068 now for qa's sections on fixture #34, with the cost estimated before any run; exec and fullsolve later. Every later cut of qa's 3,845 lines waits on this measurement, so that the cut is chosen by data. Fixture #34 is the one fixture with headroom: /qa named 1 of 5 planted defects in 8 of 8 runs. Its ground truth lives only in admarble/smpl-sequant (private), and the held-out fixture #1272 names stays unread while sections are chosen. Founding doc P-10, OQ-4.

Decisions

ID Decision Rejected
P-1 The pre-registration is experiments/E8-qa-leave-one-out.md in admarble/smpl-sequant, under that repo's rules (committed before any scored run; amendments dated) sequant's docs/investigations/ (ground truth would sit beside it on a public repo)
P-2 Sequant gets one comment on #1068 with the commit hash, the arms, the run count and the cost estimate, carrying the marker SEQUANT_PREREG, and no fixture identifier no comment (the tracker would not know the measurement exists)
P-3 Firing it is a separate owner decision after the estimate (§10) firing on approval of this node

Acceptance Criteria

  • AC-1: The pre-registration is committed in the research repo. Evidence: git -C ~/Projects/smpl-sequant log --format=%h -- experiments/E8-qa-leave-one-out.md | wc -l | tr -d ' ' → ≥ 1
  • AC-2: It has the sections the research repo's method requires. Evidence: grep -cE '^## (Question|Arms|Fixture|Decision rule|Cost estimate|Stop rules)$' ~/Projects/smpl-sequant/experiments/E8-qa-leave-one-out.md → 6
  • AC-3: The owner approved it in the research repo's decision log. Evidence: grep -c 'E8' ~/Projects/smpl-sequant/decisions/log.md → ≥ 1
  • AC-4: #1068 carries the comment. Evidence: gh api repos/sequant-io/sequant/issues/1068/comments --jq '[.[].body|select(test("SEQUANT_PREREG"))]|length' → ≥ 1
  • AC-5: The comment carries no fixture identifier. Evidence: gh api repos/sequant-io/sequant/issues/1068/comments --jq '[.[].body|select(test("SEQUANT_PREREG"))]|last' | python3 -c 'import json,os,sys;t=json.load(open(os.path.expanduser("~/Projects/smpl-sequant/experiments/E6/fixture/ground-truth.json")));b=sys.stdin.read().lower();print(sum(i.lower() in b for d in t["defects"] for i in d["identifiers"]) if b.strip() not in ("","null") else "no-comment")' → 0

isGateTestEvidence classifies AC-2 and AC-5 as gates (parse-check, 2026-11-04). This node changes nothing in sequant's tree and runs no /qa phase (Execution: runner), so §6i never reads them and no SEQUANT_MUTATION record applies; the runner runs each command and pastes its output on the issue.

May touch

admarble/smpl-sequant: experiments/E8-qa-leave-one-out.md (new), decisions/log.md, journal.md. sequant-io/sequant: one comment on #1068.

Non-Goals

  • Running it (§10), choosing sections to cut, exec and fullsolve (D-4: later).
  • Any read of #1272's held-out fixture.

Done when

A committed, owner-approved pre-registration with a cost estimate exists, and #1068 points to it without naming the fixture's content.

Lingua principale
TypeScript
Stelle
1
Fork
1
Merge medio
11h 3m
PR unite (30g)
84

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sequant-io/sequant

Tutte le issue di sequant-io/sequant

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.