Experiment: verified against unverified auto-curation

Open
  • Is something wrong?
#253 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
machine-learning, testing-qa

Research direction

Start with evals/REPORT.md, especially §§2b, 4m, and 5, then inspect the existing A/B harness and the store's --run/--expect verification. Build and compare the four arms, measuring reproduction rate, fabricated lessons on the control pool, and briefing size. Done means reporting the results and each arm's fabricated lessons; the issue does not name specific harness files or tests.

Written by the indexing model from the issue text.

Description

Why

The 2026 literature on self-evolving agent memory admits new lessons by the agent's own judgment: ReasoningBank (arXiv 2509.25140) distils memory from successes and failures "judged by the agent itself without ground-truth labels", and ACE (arXiv 2510.04618) curates playbooks from execution feedback. Both report gains. Phantom Guardrails (arXiv 2607.13083) shows the same loop inventing rules for failures that never occurred. Nobody has published the comparison that decides between them: does admitting a lesson only when its check ran red on the old state and green on the new one keep the gains and remove the fabrication?

okl is the right instrument. The A/B harness, the judge-not-generator rule and the receipt discipline in evals/REPORT.md already exist, and the store's --run/--expect verification is the admission rule under test. Running this deliberately revisits #26 (no record comes from a running system): the proposer here is runtime input, admitted only through verification.

Arms

  1. Human-recorded: the current seed store, as the existing instrument measures it.
  2. Unverified auto-curation: lessons proposed by a reflector after each task and admitted as written. Use the released ReasoningBank code where it fits the harness, otherwise an in-house reflector with the same admission rule.
  3. Verified auto-curation: the same proposals, admitted only when the proposal names a check that fails on the pre-fix tree and passes on the post-fix tree.
  4. Control: a featureless pool, after Phantom Guardrails' Counterfactual Fabrication Lab, where the correct number of new lessons is zero. Report fabricated lessons per arm.

Measure

Reproduction rate on the harness (REPORT §5 finding 7 is the pooled figure on the documented instrument; a new series runs on §4m's instrument and sets its own baseline, per §2b), fabricated lessons on the control pool, and briefing size per arm. Memory Transfer Learning (arXiv 2604.14004) says abstract insights transfer and raw traces hurt, so proposals are recorded at the Rule/Defect level, never as traces. Related: #97 (candidate inbox) is where unverified proposals would wait in production.

🤖 Generated with Claude Code

Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 17m
Merged PRs (30d)
128

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from emeraldleaf/okl

All issues in emeraldleaf/okl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.