[feature request] ISA and Assay never require that a passing eval was seen failing, and the…

Open
#2,623 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 5 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript

Research direction

Start by reading skills/Hardening/SKILL.md, ISAFormat, and the ISA and Assay skills; the issue says these currently lack mutation testing and golden set safeguards. The proposed work spans ISA frontmatter, eval behavior, and grading by a separate agent, so first determine where those rules and checks belong. Done means golden sets are pinned and checked, a recorded mutation probe turns evals red, and held-out cases are graded by a different agent or model.

Written by the indexing model from the issue text.

Description

What happened

ISA and Assay never require that a passing eval was seen failing, and the building agent can edit the golden cases. Pin the golden set by hash, require a mutation probe with a recorded red run, grade held-out cases by a different agent.

Summary

ISA criteria and Assay evals can pass without ever having been seen failing, and nothing stops the agent that builds the work from editing the expected cases it is graded against. An agent can therefore "pass" its own tests.

Proposal, three small rules for ISAs whose criteria are judged against expected cases:

  1. Pin the golden set before the first build commit: golden_set and golden_set_sha256 in the ISA frontmatter, checked before close.
  2. At least one Test Strategy row is a mutation probe: a deliberately wrong output (wrong verdict, quote absent from the source) must turn the evals red, and that red run is recorded before the criterion closes.
  3. Held-out cases are graded by a different agent or model than the one that built.

Evidence: skills/Hardening/SKILL.md line 17 says mutation testing is not built; neither ISAFormat nor the ISA and Assay skills mention a pinned golden set or a recorded red run.

I run this locally as a user overlay plus a small checker (hash, check, upstream), with 6 tests that each turn red when a guard is removed. Happy to open a PR against ISA and Assay if this direction fits.

Details

  • Kind: feature request
  • LifeOS version: 8.761.0

Filed with the LifeOS /feedback command. The reporter approved this exact text before it was sent; paths, secrets and personal details were stripped locally.

Dominant language
TypeScript
Stars
19.9k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from danielmiessler/LifeOS

All issues in danielmiessler/LifeOS

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.