CLI restructure: nine task-oriented groups over the existing 64 operations (proposal and plan)

Start by reading the CLI registration code and the handlers, artifacts, docs, and tests named in the audit, then compare those current contracts with the proposed target tree and open questions. The issue is a proposal for review; several target commands require new contracts and code, and implementation depends on agreement about the design decisions. Done means the decisions are resolved and follow-up implementation work has a clear agreed scope.

Why

dokimi has 25 top-level commands, plus the product alias. They hold 64 leaf operations. The roots grew one feature at a time, so the tree describes where code was added, not what a user wants to do:

• Names that suggest more than the command does. run sounds like the main evaluation; it is a docs lint. compare diffs those lint snapshots, not eval results. evaluate (the pipeline) and eval (scoring) sit next to each other.
• One operation in several places. Execution is spread over answers, arms, benchmark, evaluate, subjects, tasks, explore and repo run. Comparison is spread over compare, baseline compare, report model-compare, arms plugin-delta and repo compare. Reports are spread over report, dashboard, repo report and novelty view.
• Names that promise something the code does not do. personas approve checks fields; no approval is recorded and nothing checks for one. catalog trigger-set creates an eval set (and can call an LLM); it does not browse a catalog. report model-compare runs paired tests and Holm/BH correction inside a command that reads as rendering.

This issue proposes an organisation around domain objects, evals and experiments. It is backed by a code audit and three small experiments (below). It is meant for review and discussion first; implementation PRs come after we agree on the decisions in "Open questions". No new command is implemented yet.

---

1. Proposed structure: nine groups

dokimi
  skill        Work on a skill, and skill-specific diagnostics
  repo         Prepare and diagnose a repository
  docs         Static checks of documentation

  evals        Build evals, check their quality, and score
  experiment   Plan and execute experiments
  runs         Results, comparisons, reports and saved references

  tools        Tool-use probes
  trace        Analysis of an agent's execution
  data         Import, export and catalogs of inputs

Group	Question it answers	Why it is its own group
skill	What should change in this skill?	Skill-specific methods: content novelty, routing, shortening
repo	What does an agent need in this repo, and is the guidance reachable?	Revisions, environment, manifests, reachability of instructions
docs	What is wrong with the documentation?	Static checks that also apply outside a repo
evals	What do we measure with, and how far can we trust it?	Tasks, graders, judges and their quality checks
experiment	What do we run, against what, under which conditions?	Variants, models, environments, repetitions, budget
runs	What happened, and what decision do the results support?	History, artifacts, comparisons, reports, gates
tools	How does the agent choose and use tools?	Today: preference. Later: other tool probes
trace	How was the work done?	Process analysis, independent of what was evaluated
data	Where do inputs come from, and where do results go?	Catalogs, import, export

Design rules

1. Domain groups hold specialised operations; generic operations are not duplicated per group. Model runs, comparisons and result management are written once, in experiment and runs. Example: repo reach stays in repo, but a full A/B of two repo variants goes through experiment run and runs compare.
2. One interface does not mean one method. Executors keep their own result types and statistics: judge scores (answers/arms), pass rate with McNemar (tasks), gated grades with sign-flip tests (repo), localization F1 (explore). What is shared is the envelope, meaning provenance, cost and how a result was produced. The verdict logic is not shared.
3. Scoring methods stay separate commands. evals score pointwise|pairwise|panel|checklist|grounding|ragas is not folded into score --method X: their inputs and limits differ.
4. A shared catalog interface does not mean a shared schema. Products, subjects and skill catalogs are four different file formats today. data catalog addresses them uniformly with --kind; each kind keeps its own schema and its own show renderer.
5. A transformation produces a candidate, not a claim. skill shorten writes a candidate. Whether it is better is settled by a separate experiment.

Target tree

dokimi skill     novelty {analyze, atoms, probe, items} · routing · shorten
dokimi repo      fetch · validate · scan · reach
dokimi docs      check · compare(?)
dokimi evals     personas {discover, validate}
                 questions {generate, analyze, refine, review}
                 triggers {generate}
                 tasks {list, check}
                 score {pointwise, pairwise, panel, checklist, grounding, ragas}
                 judge {validate, wording, agreement}
dokimi experiment  validate · estimate · plan · run · batch · executors
dokimi runs      list · show · resume · compare · report · gate · baseline {save, list}
dokimi tools     preference
dokimi trace     inspect
dokimi data      import · export · catalog {list, show, validate, intents}

experiment plan|estimate|run --plan, and runs list|show|resume|gate, are new contracts. They need new code, not a rename (see §4).

---

2. What the code audit found

The audit covered all 64 leaves, the registration code, the artifacts, and the docs and tests. Most of the proposal's claims hold; the corrections are marked ⚠.

What the commands do

Command	Finding
run → docs check	⚠ Not purely static. With example_pass_rate in the spec, it executes the docs' code blocks in a subprocess or docker (runner/run.py:76-108, metrics/example_runner.py). It also writes baselines/current.json, which collides with the baseline group's baselines/eval/.
compare	Diffs only the summaries of run snapshots (runner/compare.py:26-48). Snapshots have no schema_version, so a generic runs compare cannot recognise them.
personas approve	Field check only (commands/personas.py:73-98); nothing is written and nothing reads an approval. Renaming it to validate is correct.
catalog trigger-set	Builds a labelled query set; with --rewrite it calls an LLM (catalog.py:65-142). It belongs in evals triggers generate.
catalog scan	A loader check over skills.yaml/SKILL.md and suites.json/task dirs; exits 1 on failure. data catalog validate fits.
report model-compare	Pairs per question, runs paired tests, Holm/BH and replicate pooling (report/model_compare.py:478-654,1077). This is a comparator inside a reporting command.
report eval vs report generate	They give different numbers on the same file. eval counts only questions scored in both arms and runs paired tests; generate averages the arms separately with no test. This is already tracked as #4 wave 4 (eval/compare.py).
novelty view	Render only, no LLM.
novelty shorten	Writes the best-safe candidate (--out). It claims knowledge is retained, not that the skill improved. Consistent with rule 5.
novelty routing	Standalone; needs no atoms or probes. skill routing fits.
novelty items	⚠ Not standalone: needs ≥3 probes.json files from novelty probe on one artifact revision. Keeping it under skill novelty is right.
context trace	Parsing only, no quality score. The only accepted format is Claude Code stream-json. trace inspect fits.
repo tasks check	Checks task quality: the untouched workspace fails the grade, the oracle passes, the isolation probe passes. This is eval quality, so evals tasks check is right. Repo readiness is repo scan.
library / subjects / catalog	⚠ Four file formats with no common schema; subjects list errors without --subjects-root. --kind is workable; show must render per kind.
tasks list / explore list / repo tasks	⚠ Three task formats (Harbor/BenchFlow TaskSpec, ExploreTask YAML, task.json under repo-eval.yaml). evals tasks list --type is a dispatcher; there is no common schema behind it.
Dead code	report.py:82 cmd_report is never registered. Hints in personas.py:70,100, baseline.py:143,170, model_compare.py:916 and datasets_export.py:147 still say python cli.py ….

Execution commands

• Two answering stacks. evaluate/benchmark use orchestrator.EvaluationPipeline → Answerer + Judge. arms/subjects use ArmRunner. A recipe cannot unify them until one stack serves both.
• Composition today:
  • benchmark run and benchmark batch → EvaluationPipeline (they do not call evaluate).
  • subjects run builds an argparse.Namespace and calls cmd_arms_matrix_run (subjects.py:138). It then runs TaskSuiteRunner, which is what tasks run uses.
• Closest existing plan formats.
  • The subject descriptor (subjects/models.py) holds suite, products, questions, tasks, baseline and matrix.cells.
  • matrix.cells JSON (eval/cells.py:147) has no schema file.
  • Neither has a plan schema under dokimi/schemas.
• Resume. Only repo run has cell-level resume: it skips finished cells, holds a lock and pins the batch to tree/eval/image (repos/runner.py:148). The pipeline reuses personas/questions only. Arms has only the opt-in DOKIMI_RESPONSE_CACHE. There is no --resume flag anywhere.
• Budget. Spend caps exist only in repo (--budget, --max-batch-usd), novelty (--max-usd) and tools (--max-cost-usd), each with different semantics. Nothing estimates dollars before a run. What exists: novelty --estimate-only and tools --dry-run count calls, repo reach --dry-run lists the planned runs, and benchmark preflight validates only.

Artifacts and provenance

• No common envelope.
  • save_artifact stamps only schema_version.
  • Run id, argv/config, git sha, dependency versions, start/end time and cost are missing, or present in only some kinds under different keys (generated_at vs evaluated_at).
  • build_run_manifest (git sha, dirty flag, deps, spec hash) is used only by the static run.
• No run registry. Outputs land in results/<kind>/, repo-runs/, reach-out/, repo-check/, . or a path the user chose. runs list has nothing to list today.
• Writers that bypass save_artifact (unversioned output):
  • panel eval
  • repo result.json
  • novelty side files: probes, routing, atoms, items
  • judge-reliability reports
  • ragas and grounding outputs
  • dashboard
  • baseline copies
• Gates are pure functions (gate/score_gate, critical_bands, regression, trust_gate, novelty_gate). A standalone runs gate is feasible once it can dispatch on schema_version.
• 18 places pair, diff or test results. Wilson and bootstrap are each implemented twice (repos/report.py:125, novelty/measure.py:526 vs eval/stats.py). Plain mean/delta code is repeated in baseline, generator and dashboard.
• Exit codes. The convention is 0 ok, 1 verdict fail, 2 input error (cli.py:97-101). baseline breaks it: a missing file exits 1.

---

3. Experiments

E1: can the new tree be built without touching handlers? Yes, 64/64.

Prototype (scratch, not committed): build the current parser, then remount every leaf under its proposed path with add_parser(name, parents=[old_leaf]). Check that the handler (func), the option set and the defaults are identical, and that --help renders.

remounted 64 leaves under roots ['data','docs','evals','experiment','repo','runs','skill','tools','trace']; mismatches=0
old leaves covered: 64 / 64

This works because dispatch goes only through args.func; no handler reads args.cmd or *_cmd. To fit the 23 leaves that merge into experiment/runs, the executor, comparator or renderer becomes a subcommand, for example experiment run repo …, runs compare repo …, runs report eval ….

Consequence: the rename can be its own, purely mechanical stage, separate from the new contracts.

E2: does one flag surface for experiment run make sense? Not as one flat flag set.

Flags of the 10 execution commands (answers generate, arms run|matrix-run, evaluate, benchmark run|batch, subjects run, tasks run, explore run, repo run):

94 distinct flags; 57 appear in exactly one command
shared by ≥5 of 10: --model(10) --provider(7) --max-tokens(6) --judge-model(6) --judge-provider(6)
                    --concurrency --context7-id --rerank-threshold --top-k --out-json --out-md --output-dir (5)
repo run: 28 flags, 26 unique to it

The same concept is spelled differently across commands:

Concept	Spellings
Output	--out, --output, --out-dir, --output-dir, --results-dir, --out-json/--out-md
Arms	--arms a,b (CSV, 4 grammars) vs --arm (repeatable, repo)
Models	--model single vs repeatable (repo) vs matrix.cells vs --work-model
Repetitions	--multi-run, --repeats, --work-repeats, --reps
Harness	in arms only a stamped label (anything but arms-runner is refused), in tasks --harnesses it actually runs, in repo it is a profile
Budget	--budget is USD in repo but --budgets is token fractions in novelty; spend caps are also spelled --max-usd, --max-cost-usd, --max-batch-usd
--max-tokens	retrieval tokens in answers/benchmark, output tokens in arms/novelty, per-call cap in tools
Concurrency	--concurrency vs -j/--jobs (repo); evaluate hard-codes 5

About 101 add_argument calls declare model/provider/judge flags; there is no package-wide helper.

Consequence: experiment run --type X would show a union of about 94 flags in --help, and argparse cannot make flags conditional on --type. Use a positional executor: experiment run <executor>. Each executor gets its own flags, plus a shared vocabulary (§4, B2). A plan-driven experiment run --plan plan.yaml comes later and sits on top of the executors.

E3: can deprecated paths stay working and hidden? Yes.

Old paths can be registered without help=, with a metavar listing only the new roots, and with func wrapped to print a deprecation line to stderr. They then parse and run, but do not appear in --help or in the generated CLI reference. Existing aliases are the only precedent (library → product, with no warning); there is no deprecation mechanism yet.

E4: the three walkthroughs on paper

Walkthrough	Today	Proposed	Gap found
Improve a skill	novelty analyze → novelty shorten --out → arms run --arms none,skill:A,skill:B --judge → read arms md	skill novelty analyze → skill shorten → experiment run arms (variants) → runs compare → runs report	Steps are chained by hand-copied paths. The shorten candidate has to be put into a skill dir to be usable as a skill:<path> arm. The candidate carries no provenance (source revision, atoms used). Plans (§4, D1) should bind these steps.
Check AGENTS.md	All in repo: scan → reach → tasks check → run --arm guidance:none --arm guidance:raw → compare → report	repo scan → repo reach → evals tasks check → experiment run repo → runs compare repo → runs report repo	⚠ A flow that today is coherent in one group is split over four. The split is correct (rule 1), but it needs a documented route: an epilog in repo --help plus a docs page, "Evaluating a repository". Keep the old repo run/compare/report paths as deprecated aliases for at least one minor release.
Cheapest adequate model	arms matrix-run with one cell per model → report model-compare (it pools cost_summary)	experiment plan (models × fixed judge) → experiment estimate → experiment run → runs compare → runs gate --min … → choose	Cost is recorded for arms only (cost_summary); eval, explore and the top level have none. Nothing estimates dollars up front. Nothing encodes "adequate" as a rule. Needs the envelope cost field (B1), experiment estimate (D2) and runs gate thresholds (C3). Whether runs compare picks the cheapest passing model or only reports cost-vs-quality is an open question.

Result: the nine groups hold up. The real work is in the contracts underneath (envelope, registry, flags, comparators, plans), not in the names.

---

4. Plan

This plan follows the conventions of #4:

• Each PR has one purpose.
• Renames, behaviour-preserving refactors and number-changing changes are never mixed in one PR.
• Every PR touching cli.py adds a changelog.d fragment.

The project is pre-1.0 (SemVer; docs/releasing.md: renaming a command is a minor bump), so deprecations take one minor release.

Stage A: groundwork (no user-visible rename)

• A1 CLI route table and deprecation mechanism in cli.py.
  • A (new_path → handler) table, and (old_path → new_path) deprecated aliases.
  • Deprecated aliases are hidden from help and print a stderr warning.
  • test_cli_help covers aliases; test_docs_code_refs accepts aliases but flags them in docs.
• A2 Exit-code contract (0/1/2) documented as public. Fix baseline exiting 1 on a missing file.
• A3 Hygiene: remove the unregistered cmd_report; replace python cli.py hints with dokimi ….

Stage B: shared contracts (precondition for runs and experiment)

• B1 Run envelope, stamped by save_artifact.
  • Fields: run_id, kind, started_at/finished_at, argv and resolved config, dokimi_version, git sha/dirty, models, input digests, cost/usage, parent_runs.
  • Built from runner/manifest.build_run_manifest.
  • Additive: old files still load.
• B1b Route the bypassing writers through save_artifact: panel eval, repo result.json, novelty side files, judge reliability, ragas/grounding, dashboard, baseline.
• B2 Shared flag vocabulary, as dokimi/cli_flags.py. Overlaps #4 wave 5 ("shared CLI model/provider flags"); land it there.
  • Canonical names: --out-dir, --model (repeatable where several models are allowed), --reps, --concurrency, --max-usd, --dry-run, --seed.
  • Split --max-tokens into --retrieval-tokens and --max-output-tokens.
  • Old spellings become deprecated flag aliases.
• B3 Runs root and index.
  • Default results/ (configurable), with one directory per run and an append-only index.jsonl.
  • Move docs check's default output out of baselines/.

Stage C: the runs group

• C1 runs list, runs show over B3/B1.
• C2 runs compare dispatching on schema_version to comparators built on eval/stats.py and the planned eval/compare.py (#4 wave 4).
  • Absorbs baseline compare, arms plugin-delta, repo compare and the statistics of report model-compare.
  • The report eval/report generate reconciliation changes published numbers, so it is a separate PR.
• C3 runs gate: dispatches the existing gate functions over a saved artifact; thresholds come from spec YAML or flags; exit 1 on a failed verdict.
• C4 runs report: renderers only (eval, summary, repo, novelty, dashboard, model table). Statistics move to C2.
• C5 runs baseline save|list on the runs root; any artifact kind, not only eval.

Stage D: the experiment group

• D1 experiment_plan.v1 schema, generalising the subject descriptor and matrix.cells.
  • Contents: executor, variants (arms/plugins/profiles), models, judge, reps, budget, success criteria.
  • experiment plan writes it; experiment validate checks it (generalises benchmark preflight).
• D2 experiment estimate: call counts × litellm prices → USD, per executor. Reuses --estimate-only/--dry-run.
• D3 experiment run --plan: dispatches to the executors, with cell-level resume modelled on repos/runner.py; runs resume re-invokes from the stored config.
• D4 Recipes: evaluate, benchmark run and subjects run expressed as plans.
  • Parity tests first: inputs, curated questions, cache reuse, and output composition must match.
  • Blocked on a single answering stack (Answerer vs ArmRunner).
  • The old entry points are deprecated only after parity.

Stage M: the mechanical rename (runs in parallel with B, after A1)

One PR per area. Each moves its paths into the new tree, makes the old paths deprecated aliases, and updates docs/tests/CI to the new spelling.

• M1 evals (19 leaves): personas, questions, triggers, tasks, score, judge
• M2 skill, trace, tools, data (16 leaves)
• M3 docs, repo (6 leaves; repo tasks check → evals tasks check)
• M4 experiment run <executor>, experiment recipe <name>, experiment validate|batch|executors, runs compare <comparator>, runs report <renderer>, runs baseline (23 leaves)

At the end of M4 every one of the 64 operations has its new path. Stages C and D then replace the M4 subcommand forms with dispatchers; the subcommand forms can stay as the explicit low-level path.

Stage R: removal

• R1 One minor release after M4: remove the deprecated paths and the product alias.

Migration surface (measured)

Where	Size
Docs and READMEs	~700 text references to old command paths, mostly in docs/
Package code	135 backticked command hints in 31 .py files
Tests	164 build_parser().parse_args([...]) calls in 29 files, plus 16 through _parse helpers
CI and .agent-eval	4 CI workflow lines (ci.yml:87, agent-quality.yml:123-162); .agent-eval/tasks/*/check.py and its oracle patches
CLI reference	Generated from build_parser() by scripts/docs/gen_reference.py; regenerates by itself
Existing guard	tests/test_docs_code_refs.py already fails on unknown commands in docs, so M PRs cannot leave stale docs
External consumers	None found in the repo; oneDAL appears only as an example target

---

5. Full mapping (all 64 operations)

Each operation has two columns:

• Stage M: the mechanical path, proven in E1.
• Target: the path after stages C and D.

Expand mapping table

Current	Stage M path	Target	Note
run	docs check	docs check	Can execute examples (§2)
compare	docs compare (?)	docs compare or runs compare (static)	Open question Q3
ingest docling	data import docling	data import --adapter docling
dataset export	data export	data export	questions, answers, eval, arms, task_runs → jsonl/parquet/hf
personas discover	evals personas discover	same
personas approve	evals personas validate	same	No approval exists
questions generate|analyze|refine	evals questions …	same
questions panel-review	evals questions review	evals questions review --method panel
catalog trigger-set	evals triggers generate	same	Can call an LLM
tasks list	evals tasks list	evals tasks list --type terminal-bench
explore list	evals tasks list --type localization	same	Separate format
repo tasks check	evals tasks check	evals tasks check --type repo
eval score	evals score pointwise	same
eval pairwise	evals score pairwise	same
eval panel-score	evals score panel	same
eval checklist|grounding|ragas	evals score …	same
eval judge-validity	evals judge validate	same	Exit 1 on FAIL kept
eval judge-wording	evals judge wording	same
eval cross-judge	evals judge agreement	same
answers generate	experiment run answers	same, or via --plan	No scoring required
arms run	experiment run arms	same, or via --plan
arms matrix-run	experiment run matrix	experiment run arms --cells … or plan
tasks run	experiment run tasks	same
explore run	experiment run localization	same	Not coding success
repo run	experiment run repo	same	Keeps container graders, resume, network
tasks harnesses	experiment executors	same
benchmark preflight	experiment validate	experiment validate (plan or legacy flags)
benchmark batch	experiment batch	same
evaluate	experiment recipe evaluate	plan recipe	D4 parity first
benchmark run	experiment recipe benchmark	plan recipe	D4
subjects run	experiment recipe subject	plan recipe	D4
arms plugin-delta	runs compare plugins	runs compare (dispatch)
repo compare	runs compare repo	runs compare	Pairing and sign-flip kept
baseline compare	runs compare baseline	runs compare --baseline NAME	Its own pairing is replaced by stats
report model-compare	runs compare models	runs compare → runs report	Statistics move out of the report
report eval	runs report eval	runs report	Numbers reconciled (C2)
report generate	runs report summary	runs report
repo report	runs report repo	runs report
novelty view	runs report novelty	runs report
dashboard generate	runs report dashboard	runs report --over <runs>
baseline save|list	runs baseline save|list	same
novelty analyze|atoms|probe|items	skill novelty …	same	items needs probes
novelty routing	skill routing	same
novelty shorten	skill shorten	same	Writes a candidate
repo fetch|validate|scan|reach	unchanged	unchanged
tools preference	unchanged	unchanged
context trace	trace inspect	same	Claude Code stream-json only
library list|show	data catalog list|show --kind product	same
library intents	data catalog intents	same
subjects list|show	data catalog list|show --kind subject	same	Needs a default root
catalog scan	data catalog validate	same
product (alias)	—	removed

Not proposed

• No diagnostic method is removed. The goal is fewer interfaces and correct boundaries, with the same capabilities.
• No new roots for policy, ci, models or improve. Policy is first a decision rule over results (runs gate). CI calls experiment run + runs gate. Model choice is an experiment with quality and cost criteria. Improvement is an agent workflow that tests hypotheses, not one algorithm.

---

6. Open questions for review

1. evals vs eval. During deprecation, dokimi eval score (old) and dokimi evals score pointwise (new) would sit side by side and differ by one letter. Options: (a) accept it for one release; (b) name the group eval and make the old paths sub-aliases (eval score → eval score pointwise); (c) use another name (measure?). Leaning (b): fewer typos, and dokimi eval … keeps its meaning.
2. Executor as positional or flag. experiment run <executor> (E2 recommends it) or --type? The plan file names the executor either way.
3. Where static snapshot comparison lives. docs compare keeps the docs lint self-contained, and snapshots have no schema_version. runs compare (static) would need snapshots to join the envelope. Leaning docs compare.
4. Repo route discoverability. Is a repo --help epilog plus a docs page enough, or should repo run/compare/report stay as permanent, non-deprecated shortcuts?
5. "Cheapest adequate model." Should runs compare select the cheapest model that passes (--select cheapest-passing --gate …), or only report cost-vs-quality and leave the choice to the user or agent?
6. Runs root. results/ in the working directory (today's de-facto default) or a configurable DOKIMI_RUNS_DIR? Should the index be written by every command, or only by experiment run?
7. Ordering against #4. B2 overlaps #4 wave 5, and C2 depends on #4 wave 4 (eval/compare.py). Should stage M wait until wave 4 lands (it touches eval/* and report/*), or go ahead? M only edits commands/* and cli.py.
8. Deprecation length. Is one minor release enough? No external consumers were found in the repo; are there any outside it (oneDAL dev/agent_evals, internal CI)?
9. Naming consistency. Should the roots be uniformly singular or plural (evals, runs, experiment)?

Definition of done

• Every one of the 64 current operations is reachable under the new tree; the old paths warn for one minor release, then are removed.
• --help and the generated CLI reference show only the nine roots.
• Every run artifact carries the envelope (B1); runs list/show/compare/report/gate work over all schema_version kinds.
• The three walkthroughs (§3 E4) are written as docs pages, using only new commands, and pass test_docs_code_refs.