catalog granularity: decide whether skills should be split, merged, or regrouped (static check + paid partition eval)

Start with the existing catalog check, catalog trigger-set, catalog trigger-probe, and plan catalog implementations, plus efficiency.py and the referenced issues #94, #217, #331, and #413. Read the documented arm grammar and paired bootstrap approach first; the issue describes a multi-stage CLI feature with prompt generation, routing experiments, clustering, and statistical decisions. Done means static candidates appear in catalog check, the paid command reproduces recorded PR-family and dpnp fixture results, and the method is documented in the README.

Why

The question "is this one skill or several?" keeps coming up. Two examples:

• The napetrov-skills PR family (pr-review, pr-address, pr-land, pr-queue): should it be split or merged?
• The 8 dpnp-* skills in intel/skills: quickstart, migration and troubleshooting share triggers such as NotImplementedError, fallback, dpctl device and porting from NumPy.

Today there is no command for this. The answer depends on routing behaviour, not on reading the skill. An experiment on the PR family (haiku, 3 reps, about $7) showed:

Finding	Number
Current split descriptions load the right skill	0.13 recall on held-out prompts
Merged vs split with matched wording	Recall gap is gone (S' − M = +0.10 [+0.01, +0.20]). The merge's edge was description wording.
Cross tasks (need 2+ skills)	Merged 0.67–0.75 full coverage vs split 0.04–0.29. Haiku loads at most about one split skill per run.
Sibling confusion	0
Tokens per load	Merged 21.5k vs split about 5.8k

Lessons that the command must build in:

1. Always include a sharpened-descriptions arm. Without it, merge vs split confounds structure with wording.
2. Use held-out prompts. Check for a 4-gram leak against every arm's description.
3. Decide on cross coverage and sibling confusion, not on recall alone.
4. Check whether the outcome tasks are at ceiling first. The PR outcome tasks were solved without the skill.

Proposal (in the #94 v2 grammar)

Tier 1: free, part of catalog check

A granularity section is added to catalog check output (and to repo scan as a summary line). It never gives a verdict, only candidates:

• Merge candidates. Pairs with high description similarity, shared trigger clauses, or body or snippet duplication.
• Split candidates. Bodies with several section clusters that never reference each other, a long body, or many disjoint "Use when" clauses.
• The output is a hint carrying the exact catalog granularity command and its estimated cost.

Tier 2: paid, catalog granularity ROOT [paid]

dokimi catalog granularity ROOT
  --units SKILL[,SKILL...]          # merge/regroup: the units are skills
  --split SKILL                     # split: the units are the skill's sections (H2/H3 or job segments)
  --from-check                      # take candidates from catalog check
  --k auto|1..N                     # default auto: dendrogram cuts, see Method
  --arms as-is,sharpened,partition[,router]
  --prompts auto|FILE --heldout-check
  --outcome TASKS_GLOB              # optional phase 2, reuses plan catalog / eval tasks
  --model haiku --reps 3 --budget USD

• plan catalog --preset live can include it as one cell.
• Output: verdict.json plus a markdown section in report. The verdict is one of no-change | sharpen-descriptions | merge{groups} | split{k, parts} | add-router, with the deciding metric and its CI.

Method: choosing K and the grouping

Brute force is out. Partitions of 6 skills number 203, and of 8 skills 4140. Instead:

1. Units. For a merge, the units are skills. For a split, they are sections or job segments.

2. Need matrix. Generate user-voice prompts per unit, per pair (cross) and as negatives. Each prompt is labelled with the set of units it needs.

3. Pair affinity (cheap). Three signals feed it:

   • co-need rate;
   • sibling confusion from one as-is routing pass;
   • static overlap (description similarity, body Jaccard, shared snippets).

   The distance between two units is 1 − affinity.

   • Semantic validation, 11-06: embeddings (bge, Cohere v4) and TF-IDF do not predict confusion, cross-need or wording gains. Example: 6–9 pairs predicted, 0 real on the PR family. Their natural K is unstable. Static signals therefore only build the shortlist; affinity has to come mainly from measured co-need and confusion.

4. Candidate partitions. Average-linkage clustering gives N nested partitions, one per K = 1..N. The cut with the largest dendrogram gap or the best silhouette is the "natural K". Keep K = 1, K = N, the natural K and its neighbours. That is about 4 to 5 candidates.

5. Hard constraints filter the candidates:

   • every part must get a description under 1536 chars;
   • a classifier must be able to tell the descriptions apart on the prompts;
   • each part must be at least a minimum size;
   • shared invariants either stay in one part or go to a shared reference file.

6. Paid routing for each surviving candidate. It uses the sharpened arm and held-out prompts, at 3 reps.
   The objective is J = cross_coverage + recall − confusion − λ·tokens_per_load.

   • Parsimony rule: choose the fewest-tokens candidate whose J lies within the CI of the best one.
   • K = 1 wins (no split / merge all) when co-need is high, i.e. most tasks need 2+ units, or when the clustering shows no natural gap.
   • K = N plus a router is tested when the split wins recall but loses cross coverage.

7. Outcome (optional). Run the chosen partition against as-is and no skill on discriminating tasks.

Done when

• catalog check emits granularity candidates (free, warn-only in CI).
• catalog granularity reproduces the PR-family result, and the dpnp result once it is in, from a fixture with a recorded run.
• The verdict and the method are documented in the README (user docs, not AGENTS.md).

Related: #94 (CLI grammar), #217 (trigger-probe), #138 (catalog CI), #331 (eval coverage), #413 (corpus-scale scan gaps).

Flow

1. catalog check (free) returns the candidates and a hint with the estimated cost.
2. The user opts in.
3. Prompt generation, with the leak check.
4. Arm build: as-is, sharpened, partitions, router.
5. Probe-0: an as-is routing pass. It measures co-need and confusion, and from those builds the candidate partitions.
6. Routing, with paired CIs and the J score.
7. Outcome (optional). It runs a ceiling gate first.
8. Output: verdict.json and the sharpened descriptions as a patch.
9. Regression re-run on the pinned prompt set whenever the descriptions change.

Integration

Piece	Reuse	New
Static candidates	catalog check, #413	granularity section (TF-IDF overlap, duplicated sections and snippets, far-away prompts)
Prompts	catalog trigger-set, eval tasks generate	--kind granularity, labelled with units, plus the n-gram leak check
Arms	arm grammar §2.5, the skill-catalog arm	sharpened, partition and router builders
Routing	catalog trigger-probe (#217), the no-notice discover arm (#110)	capture of multiple loads, no-load recorded as its own outcome
Stats	efficiency.py paired bootstrap	J score, the parsimony rule, candidates from affinity
Outcome	plan catalog, Harbor tasks, #331	ceiling gate
Output and CI	§2.6/§2.7, #318, #138	verdict.json; in CI, static warnings only and a paid run on a label

PR order

1. Static granularity section in catalog check.
2. Prompt generator and leak check.
3. Arm builders and multi-load capture.
4. catalog granularity: probe-0, partitions, verdict.
5. Outcome phase with the ceiling gate.
6. plan catalog cell and the Action label.

Blockers

• Harbor runs with the Claude CLI fail every model call in the container (api_retry, unknown error), while a plain container reaches Bedrock. The ${AWS_BEARER_TOKEN_BEDROCK} text in the job config is Harbor's redaction for persistence and is resolved at run time, so it is not the cause. Leading suspect: the egress-control sidecar blocking bedrock-runtime.*. Next suspect: the background small/fast model resolving to a non-Bedrock ID.
• The paired-bootstrap stats are unmerged (the skill-utility branch).
• Outcome tasks sit at ceiling without the skill: dpnp Harbor no_skill scores 1.0, and the PR outcome tasks are solved without it. Discriminating tasks need to come from #331.