[feature request] ISA and Assay never require that a passing eval was seen failing, and the…

Aperta
#2,623 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

@mj-deving ci sta già lavorando.

Dal 20/11/2026.

  • #2889 di @mj-deving — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
typescript

Direzione di ricerca

Start by reading skills/Hardening/SKILL.md, ISAFormat, and the ISA and Assay skills; the issue says these currently lack mutation testing and golden set safeguards. The proposed work spans ISA frontmatter, eval behavior, and grading by a separate agent, so first determine where those rules and checks belong. Done means golden sets are pinned and checked, a recorded mutation probe turns evals red, and held-out cases are graded by a different agent or model.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happened

ISA and Assay never require that a passing eval was seen failing, and the building agent can edit the golden cases. Pin the golden set by hash, require a mutation probe with a recorded red run, grade held-out cases by a different agent.

Summary

ISA criteria and Assay evals can pass without ever having been seen failing, and nothing stops the agent that builds the work from editing the expected cases it is graded against. An agent can therefore "pass" its own tests.

Proposal, three small rules for ISAs whose criteria are judged against expected cases:

  1. Pin the golden set before the first build commit: golden_set and golden_set_sha256 in the ISA frontmatter, checked before close.
  2. At least one Test Strategy row is a mutation probe: a deliberately wrong output (wrong verdict, quote absent from the source) must turn the evals red, and that red run is recorded before the criterion closes.
  3. Held-out cases are graded by a different agent or model than the one that built.

Evidence: skills/Hardening/SKILL.md line 17 says mutation testing is not built; neither ISAFormat nor the ISA and Assay skills mention a pinned golden set or a recorded red run.

I run this locally as a user overlay plus a small checker (hash, check, upstream), with 6 tests that each turn red when a guard is removed. Happy to open a PR against ISA and Assay if this direction fits.

Details

  • Kind: feature request
  • LifeOS version: 8.761.0

Filed with the LifeOS /feedback command. The reporter approved this exact text before it was sent; paths, secrets and personal details were stripped locally.

Lingua principale
TypeScript
Stelle
19.9k
Fork
2.5k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di danielmiessler/LifeOS

Tutte le issue di danielmiessler/LifeOS

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.