[feature request] ISA and Assay never require that a passing eval was seen failing, and the…
Maintainers usually reply within 5 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- typescript
- Domain
- ai-infra-agents, testing
Research direction
Start by reading skills/Hardening/SKILL.md, ISAFormat, and the ISA and Assay skills; the issue says these currently lack mutation testing and golden set safeguards. The proposed work spans ISA frontmatter, eval behavior, and grading by a separate agent, so first determine where those rules and checks belong. Done means golden sets are pinned and checked, a recorded mutation probe turns evals red, and held-out cases are graded by a different agent or model.
Written by the indexing model from the issue text.
Description
What happened
ISA and Assay never require that a passing eval was seen failing, and the building agent can edit the golden cases. Pin the golden set by hash, require a mutation probe with a recorded red run, grade held-out cases by a different agent.
Summary
ISA criteria and Assay evals can pass without ever having been seen failing, and nothing stops the agent that builds the work from editing the expected cases it is graded against. An agent can therefore "pass" its own tests.
Proposal, three small rules for ISAs whose criteria are judged against expected cases:
- Pin the golden set before the first build commit:
golden_setandgolden_set_sha256in the ISA frontmatter, checked before close. - At least one Test Strategy row is a mutation probe: a deliberately wrong output (wrong verdict, quote absent from the source) must turn the evals red, and that red run is recorded before the criterion closes.
- Held-out cases are graded by a different agent or model than the one that built.
Evidence: skills/Hardening/SKILL.md line 17 says mutation testing is not built; neither ISAFormat nor the ISA and Assay skills mention a pinned golden set or a recorded red run.
I run this locally as a user overlay plus a small checker (hash, check, upstream), with 6 tests that each turn red when a guard is removed. Happy to open a PR against ISA and Assay if this direction fits.
Details
- Kind: feature request
- LifeOS version: 8.761.0
Filed with the LifeOS /feedback command. The reporter approved this exact text before it was sent; paths, secrets and personal details were stripped locally.
- Dominant language
- TypeScript
- Stars
- 19.9k
- Forks
- 2.5k
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from danielmiessler/LifeOS
-
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
danielmiessler/LifeOS#2894 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
danielmiessler/LifeOS#2891 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
danielmiessler/LifeOS#2888 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
danielmiessler/LifeOS#2886 ·
Maintainers usually reply within 5 days
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
danielmiessler/LifeOS#2848 ·
Maintainers usually reply within 5 days
All issues in danielmiessler/LifeOS
Similar issues
-
ready-for-agent
Difficulty 1/5 Under an hour Newbie friendliness 86/100
LucasSantana-Dev/Lucky#3257 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
XiaomiMiMo/MiMo-Code#2849 · 1 comment ·
Maintainers usually reply within 1 day
-
enhancement frontend good first issue help wanted testing-infrastructure
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 2 days
-
enhancement follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
obsidian-tasks-group/obsidian-tasks#4155 ·
Maintainers usually reply within 1 day