Skill auto-improvement: measurement gates (A) and improve command (B)

Read the README and inspect the existing eval and command structure in dokimi first; the issue names no source files or tests. Stage A is split into six small PRs, each with tests, README documentation, and a worked example on an intel/skills skill. Start by choosing one measurement component and trace how evaluations are currently run; done means its tests and documentation pass and the example demonstrates the gate or report.

Summary

Land skill auto-improvement support in dokimi in two stages:

• A — measurement building blocks (ship now): generic gates that make any skill/eval change trustworthy.
• B — improve command (after A): propose a candidate skill (GEPA / edit search) and stop at a sealed-set gate. Never writes to the skill repo.

A fully autonomous loop with auto-PRs (C) is out of scope until B is validated.

Why

Three experiment rounds on Haiku (mcr, tdg, dpnp-troubleshooting, onetbb, xpu-port, xpu-profile-unitrace, xpu-runtime-preflight) showed:

• Round-1 gains (+.13 to +.29) shrank to CIs touching 0 on fresh evals. 16–24 item evals cannot see +.1.
• Dev winners are inflated (dpnp +.35 dev → +.10 sealed). Editing against the test set inflates by ~half the edit size (simulation).
• Verifier holes were found by red-team every round: lost structural checks, half-array sort passing, 6-digit precision hiding wrong output, keyword stuffing passing 23–35% of items.
• Contaminated / ceiling evals: answer shipped inside skill; onetbb at ceiling with no skill even after hardening.
• GEPA on new skills with 48-item two-writer evals: xpu-port +.28 [+.06,+.50], unitrace +.33 [+.14,+.53], preflight +.14 [−.02,+.30]. Part of gain may be reflector-added facts (fact-check in progress).
• A shipped v0 skill was harmful vs no skill (unitrace) — candidate-vs-v0 gates alone miss this.
• Multi-model: Haiku-only selection showed no measurable harm to Sonnet/GPT-5.5; full multi-model selection costs ~25×.

Every A component below caught at least one false gain.

Stage A — measurement building blocks

Each item is one small PR.

1. Ceiling / floor detector — flag tasks where the no-skill arm ≥ .9 or all arms score 0; mark n<3 provisional. Run on every eval refresh.
2. Verifier red-team — canaries (reference answer must pass), planted wrong answers (must fail), empty/hedge answers, keyword-stuffing probe. Report pass rates; block optimization when any attack passes.
3. Leak gate + split lock — 8-gram overlap between skill text and eval items/answers; dev/val/sealed splits; sealed set records when it was scored and is marked burned if a skill is edited after.
4. Gate report — paired bootstrap 95% CI for: v0 vs no-skill, candidate vs v0, confirmation on one second model (≥3 reps), token/word cost. Show per-writer split and non-stuffable subset.
5. Coverage report — static LLM map of skill requirements vs eval checks, listing uncovered requirements. Labelled "precision unverified" until calibration lands (recall ~.8–1.0, precision ~.37 so far).
6. Multi-writer eval generation — items from ≥2 writer models that see only the skill description, not SKILL.md; recommend ≥40 items.

Acceptance: each command has tests, docs in README, and a worked example on one intel/skills skill.

Stage B — dokimi improve (optional plugin)

• Input: skill, eval with dev/val/sealed, budget.
• Candidate sources: GEPA reflection, trace contrast (with/without skill), removal/compression. Choice pending ablation results.
• Select on val; score sealed once; emit candidate + Stage A gate report.
• Fact-check step: every new literal fact added by the reflector is listed and must be verified (other model family or upstream docs) before the candidate passes.
• Optimizer dependency (gepa) stays optional (extra), not core.
• Never edits the skill repo; output is a patch + report for a human.

Acceptance: on ≥3 skills, improve reproduces a gain whose sealed CI excludes 0, with gate report attached.

Open questions

• Recommended eval size × reps for +.1 detection (power study running).
• Does trace-derived editing beat GEPA / random edits (ablation running)?
• Coverage map precision after calibration against a labelled set.
• Criteria for ever enabling C (autonomous PRs).

Trace graph as an optimisation vector (proposal, first measurements)

Skill edits should also be judged on the agent's execution graph, not only the final score. Per run:

• hops: steps from start to answer (path length after collapsing consecutive repeats);
• loops: revisits of an already-seen step label (cf. dokimi repeated_tool_calls);
• must-keep: steps the verifier needs. Shortening is accepted only if every must-keep step stays present at the baseline rate, and reward stays non-inferior (paired CI).

Measured so far (model-config-recommend, 17 tasks × 3 reps per arm, existing Harbor runs, rescored; scratch skill-autoimprove/hops/e1_offline.py):

• run_planner absent → reward .08 (n=12). Present with the task's own parameters → 103/159 pass; present with wrong --device/--ctx/... → 4/33 pass. So presence is necessary and argument correctness is too.
• Within skill arms (task-centred): r(reward, turns) −.27, r(reward, hops) −.19, r(reward, loops) −.08. Longer runs fail more, but this is confounded (failing runs may take more turns), so it is not evidence that shortening helps.
• t4-v1 vs v0: hops 3.7 vs 5.0, reward .73 vs .43 (dev+test pooled, 3 reps).

Causal test (done, 2026-10-28): t4c-v1 = t4-v1 minus 8 reference sections (133 vs 323 lines), same 17 tasks, 3 reps. The Harbor job was cancelled at the 30 min limit: 48/51 trials finished (3 cancelled: mcr-unknown-b570, mcr-unknown-b60dual, mcr-remote-code-b60, 1 rep each). Trace graph rescored with the current verifier (exp/mcr-t4/graph-t4c/trials.json).

arm	dev reward (n)	test reward (n)	hops dev / test	explore_skill per run	must-keep (planner + answer)
v0	.46 (24)	.41 (27)	5.0 / 5.0	.73	1.00 / .96
t4-v1	.79 (24)	.67 (27)	3.5 / 3.9	.04	.92 / .78
t4c-v1 (compressed)	.83 (23)	.60 (25)	3.7 / 3.9	.12	.83 / .84

Findings:

• Compression did not shorten the path (hops unchanged). Section length is not what drives hops.
• The hop drop v0 → t4-v1 comes from one removed loop: explore_skill (grep/cat/--help on SKILL.md and scripts) fell from .73 to .04 per run. The gate should credit that kind of change, not content size.
• Reward difference t4c-v1 vs t4-v1 on test is within noise at 3 reps (per-task swings 1.0 ↔ .67 on several tasks). No significant loss.
• Must-keep is not violated by compression: run_planner present 1.33 vs 1.39 calls per run.
• Failing checks are the same set for both arms (test_no_plan_for_out_of_scope_request dominates), so no check was lost to the cut sections.

Not done: coverage map (trace_graph_t4.py needs litellm, not installed here; skipped, no LLM spend).

Planned tooling: trace_graph emits hops, loops, must-keep rate per trial and a gate table (candidate vs v0). Open: how many tasks × reps are needed for a −1 hop difference to be visible.

Gate (prototype, scratch hops/gate.py, 2026-10-28)

Paired by task (per-task means, bootstrap over tasks, 10k resamples), baseline v0, 17 tasks × 3 reps. Trials re-traced with the answer-label fix (a bash heredoc into /app/answer.md is write_answer, was mislabelled check_answer).

• reward non-inferior: 95% CI lower bound of (cand − base) > −0.10
• must-keep kept: lower bound > −0.05. A step is required on a task if (a) every passing baseline run has it and (b) runs without it pass ≤ 50%. Reading/exploration steps are never required.
• hops shorter: point estimate < 0 and CI upper bound < 0

candidate	reward diff [CI]	hops diff [CI]	must-keep diff [CI]	verdict
t4-v1	+.29 [+.10, +.49]	−1.3 [−1.6, −1.0]	0	ACCEPT
t4c-v1	+.28 [+.13, +.44]	−1.1 [−1.5, −0.8]	0	ACCEPT
g2-v3	−.04 [−.22, +.14]	−.3 [−.5, −.1]	−.06 [−.12, 0]	REJECT: reward; 3 runs wrote no answer
t4b-v1	+.14 [−.04, +.33]	−.3 [−.7, +.1]	−.02 [−.06, 0]	REJECT: hops not significant; 1 run wrote no answer

What the gate shows:

• It accepts the change that removed a real loop (explore_skill), and it does not credit t4c's compression with any hop gain (its hops fall only because t4-v1's loop removal is inherited).
• A strict must-keep (any step present in all passing baseline runs) wrongly rejected t4-v1 for skipping manual_calc, which was not required (absence passed 100%). Rule (b) fixed this, so keep it.
• The first run of this gate rejected t4c on one "missing answer" run. That was a labelling bug: the run wrote /app/answer.md via a bash heredoc. Fixed, now ACCEPT. The gate caught a real labelling error only because the run was inspected.
• Margins (0.10 reward, 0.05 must-keep) are judgement calls. Hop CIs are wide at 17 tasks × 3 reps.

Next: more reps on the borderline candidates (t4b-v1 hops CI crosses 0), then decide whether to implement this inside trace_graph proper.