Repository evaluation: move the eval mechanics out of oneDAL dev/agent_evals into dokimi

This is a multi-PR design tracker, not a single task: start with the D1 repo compare item and the framework code behind repo report in the dokimi repository. Each step is independent, so pick one with a concrete scope, such as the sign-flip permutation test for D1, and check with maintainers first. Done means the step's PR lands with tests that match the acceptance criteria listed in the issue.

Repository evaluation today has two implementations of the same mechanics. The framework side (dokimi repo validate | tasks check | run | report | scan) prepares workspaces, applies arms, runs agents and graders in containers, and gates results. The first repository to use it, uxlfoundation/oneDAL#3818 (dev/agent_evals, not merged), still carries its own copy: workspace prep, self-check, the repo_grade.v1 writer, process metrics, generic graders and a guidance claim checker, about 1.8k lines of Python, kept so the tasks run without the framework.

oneDAL reviewers asked why a mostly generic harness lives in their codebase. The answer is that it shouldn't: the tasks belong with the code they test, the mechanics belong here. This issue tracks moving the mechanics into dokimi so that a repository keeps only its evals.

Target split

oneDAL dev/agent_evals/              (after)
├── repo-eval.yaml                   manifest: guidance globs, image, defaults for built-in graders
├── Dockerfile, bin/bazel            the environment tasks build in
├── tasks/<id>/                      task.json, patches, hidden tests, review gold   (data only)
├── graders/build.py, make.py        the only graders that know oneDAL
└── README.md
.github/workflows/agent-evals.yml    ~30 lines calling dokimi's reusable workflow

dokimi                               (gains)
├── repo compare                     arm vs arm, paired by task
├── dokimi:* graders                 review seeds, review replay, hidden tests, mutation
├── task `host` requirements         unsupported instead of a false pass
├── process metrics                  from events.jsonl, plus manifest-declared counters
├── L0 parity                        the remaining oneDAL static_check rules
├── arms                             guidance from another revision, patch arms
└── reusable CI workflow             validate, scan, tasks check

Decided: oneDAL drops its standalone mode, and #3818 is trimmed to evals-only before it merges, so there is no later deletion PR.

Steps

Each step is one PR based on main. They are independent unless noted.

• D1. repo compare (#42, merged): per-task paired deltas between a baseline arm and treatment arms over one or more batches, with a sign-flip permutation p (exact at reps=1, where it equals McNemar's exact test), a task bootstrap CI, Holm correction across comparisons, tasks up/down/tied, cost and turns, and any grade.json metric (--metric). Today repo report has no arm comparison; every analysis so far was an ad-hoc script.
• D2. Built-in graders, dokimi:review_seeds and dokimi:review_replay (#120, merged): task.json grader: "dokimi:<name>" is graded by framework code mounted read-only in the grading container (stdlib only); other grader names still go to the repository's grade entry point. Review tasks become pure data (manifest/gold JSON plus the answer). Ported from oneDAL graders/review.py and replay.py; dokimi's own review task is the second user.
• D3. Host requirements (#112, merged): task.json host: {min_cpus, cpu_vendors, cpu_flags}, checked against the container's view (affinity and cgroup quota). tasks check and run report the task as unsupported with the reason. Needed for a thread race that only reproduces with 3+ CPUs on Intel hosts.
• D4. Process metrics in the framework (#157): tool calls, guidance files read (from the manifest's globs), denied and failed commands, computed from events.jsonl into result.json. The manifest declares repository counters as regexes (process_metrics:, for example a bazel command count or a missing-compiler message), so oneDAL's metrics.py goes away.
• D5. dokimi:hidden_tests and dokimi:mutation (after D2): hidden files restored from a revision of {source_dir}, then test_cmd and regression_cmd templates, an optional format_cmd, root_cause, and diff size. Mutation: the agent's tests must pass and kill each mutant patch, with library sources restored from base by test_globs. Command defaults live in the manifest, not in every task.
• D6. L0 parity with oneDAL static_check.py (#115, merged): diff the claims both produce on oneDAL main and port the missing rules into the L0 adapters (Starlark macro keyword arguments, --config names from .bazelrc, and anything else the diff shows). The static entry point then becomes optional for oneDAL.
• D7. Arms beyond guidance on/off (#117, merged): guidance:rev:<sha> (the guidance of another revision on the task's base, the "does this PR's AGENTS.md help?" question, which needed generated task twins so far) and patch:<file> (tooling or layout changes).
• D8. Reusable CI workflow plus a pinned install (#113, merged): a workflow repositories call to run repo validate, repo scan and repo tasks check on the tasks a PR touches; a repository pins dokimi by tag or SHA, which the manifest records.
• oneDAL: trim #3818 (after D2–D6 and D8): delete common.py, config.py, contract.py, metrics.py, run.py, static_check.py and the generic graders; switch tasks to dokimi:* graders and host:; replace the workflow.

Parallel plan (2026-10-18)

Each wave-1 PR keeps to its own files and contract sections. Together they stay clear of the open PRs: #66 and #71 edit the contract head, §1 and the §8/§9 seam, and others edit architecture.md, README.md, AGENTS.md, docs/README.md, mkdocs.yml, ci.yml and the lines l0/adapters.py 51–58.

Acceptance for the oneDAL step

• Regrade every stored oneDAL run with the old and the new graders: verdicts and gate metrics agree on every run (the previous contract migration reached 308/308).
• dokimi repo tasks check is OK for all 27 tasks in the oneDAL image, with the host-dependent task unsupported rather than passing on hosts that cannot show its bug.
• repo scan on oneDAL main reports no fewer real failures than static_check.py.

Risks

• Grader semantics now ship with dokimi, so a regrade depends on its version: the manifest pins it, and grade.json records the grader version.
• Without standalone mode, a contributor needs docker and dokimi to self-check a task.
• Bazel details (--nocache_test_results, the output base shared by agent and grader) become manifest strings; only the regrade and tasks check catch a mistake there.
• Contract §5 ("the harness stays in oneDAL and keeps working standalone") is rewritten in D2.