[feat] Judge whether team skills load when they should, without writing a scorer

Start with src/skill-use.ts:264, src/session-flow/session-work.ts:104-131, and src/session-flow/scoring.ts:242, then trace evidence creation and session scorer resolution. The issue names unit and end-to-end coverage for capture, manifest masking and filtering, routing settings, and scoring; begin by mapping those paths to the acceptance criteria. Done means the evidence, generated routing questions, batching and on-demand scoring, and both scorer guides satisfy the listed criteria.

Parent: #408

Goal

#408's trigger quality asks whether a skill is loaded where it should be, and only there. Today a session judge cannot tell. Its evidence.json holds the subject, the kind and the gaps (judgeSession, src/session-flow/session-work.ts:104-131). It does not say which team skills the session was delivered, or which it loaded. used.skills goes on the label, but the judge never reads it.

Give the judge both, and let a team turn on routing questions for its skills in scorers/scoring.yaml. No scorer file to write, no new command.

Example

A team that judges finished sessions turns it on:

# scorers/scoring.yaml
sessions: true
skill_routing: true

A member finishes a session that asked for a review. It had the root skill review-notes and the skill release from the ops skills namespace, and loaded review-notes. The judge reads:

{
  "subject": { "platform": "claude-code", "sessionId": "7c1e…" },
  "kind": "session",
  "gaps": [],
  "skills": {
    "available": [
      { "name": "release", "namespace": "ops", "description": "Cut an npm release: bump, changelog, tag, publish. Use when asked to release or publish a version." },
      { "name": "review-notes", "description": "Write review notes for a PR. Use when asked to review a branch or a PR." }
    ],
    "used": ["review-notes"]
  }
}

One judge call answers one routing question per skill. On demand:

$ teamai score --session 7c1e… --scorer teamai:routing:ops:release --scorer teamai:routing:review-notes
ℹ teamai:routing:ops:release: idle (score 1, passed). Reasoning: ~/.teamai/scores/reasoning/4b1d….md
ℹ teamai:routing:review-notes: fit (score 1, passed). Reasoning: ~/.teamai/scores/reasoning/8e07….md

After a few weeks, the scorer view counts each skill by value (#1472):

teamai:routing:ops:release · session · 30 days
label    score   labels
fit      1       31
idle     1       112
extra    0       6
missed   0       9

precision(ops:release) ≈ 31 / (31 + 6) = 84%   loaded where it was not needed 6 times
recall(ops:release)    ≈ 31 / (31 + 9) = 78%   not loaded where it was needed 9 times

The team rewrites the description of release. It splits the view at that commit (#1182) to see whether the failed share drops, and reads missed in the counts of the weeks after it. A rewritten description stays in the same measurement.

The rubric, the same for every skill:

fit     1  <skill> is in skills.used, and the session needed what its description says.
idle    1  <skill> is not in skills.used, and the session did not need what its description says, or another skill or the agent's own steps did it.
extra   0  <skill> is in skills.used, and the session did not need what its description says.
missed  0  The session needed what the description of <skill> says, <skill> is not in skills.used, and nothing else did it.
Judge only <skill>, against its entry in skills.available.

A scorers namespace turns it off for its members, as it does sessions:

# scorers/ops/scoring.yaml
skill_routing: false

A team that wants its own rubric leaves skill_routing off and writes a scorer per skill.

scorers/ops/routing-release.md:

---
labels:
  - { value: fit, score: 1, description: "release is in skills.used, and the session needed it." }
  - { value: idle, score: 1, description: "release is in skills.available, is not in skills.used, and the session did not need it, or another skill or the agent's own steps did it." }
  - { value: unavailable, score: 1, description: "release is not in skills.available." }
  - { value: extra, score: 0, description: "release is in skills.used, and the session did not need it." }
  - { value: missed, score: 0, description: "The session needed what the description of release says, release is not in skills.used, and nothing else did it." }
passingScore: 1
samplingRate: 100
sessions: true
---
Judge only the team skill `release`, against its entry in `skills.available` in `evidence.json`.

With batch: true in scorers/scoring.yaml, a session's scorers share one judge call. Each scorer samples sessions on its own, so ten per-skill scorers at 25% still reach about 94% of sessions, with about a quarter of the answers. At 100% the same launches answer every scorer. To spend less, lower daily_limit, not samplingRate.

Scope

Evidence

• At session start, the capture records the frontmatter description of each delivered team skill, read from the same SKILL.md it already checks against the delivery record (deliveredTeamSkills, src/skill-use.ts:264), and the skill's namespace. The session's own pull adds the skills it delivers with the description they then have, as it adds their files. A later pull changes nothing.
• The namespace is the skills namespace pull delivered the skill from, for the namespaces active at that capture. collectTeamRepoSkills keeps only the first namespace of a name, and the delivery record is path to hash, so neither gives it alone.
• skills.available lists the team skills the session's captures recorded under the session tool's skill roots and the shared roots. A copy installed only for another tool is not listed. When captures differ, the list is their union and the latest description wins. A root skill has no namespace.
• skills.used is the label's used.skills, unchanged: bare names.
• Descriptions are masked with the scope's mask before they go into the manifest, and count toward the evidence cap as the manifest already does.
• A capture that read the delivery record and found no team skills gives skills: { available: [], used: [] } and no gap. When no capture of the session recorded descriptions, for example a capture written before this change, there is no skills key and the evidence gets a new gap, skills_unknown.
• The control of a tried scorer file (#1399) gets the session's own captures instead of none (src/session-flow/scoring.ts:242), so it keeps skills.available. Its skills.used holds only skills the first prompt itself invoked. Without the list, a per-skill scorer passes the control as unavailable on almost every try.

Routing questions

• skill_routing is a key of scorers/scoring.yaml, off by default, and resolves as sessions does: an active scorers namespace's scoring.yaml overrides the root file, and one that sets false turns it off. It acts only where session judging is on. There is no member key in config.yaml.
• A session with routing on gets one question per skill in its skills.available, and none for any other skill. A session with skills_unknown gets none.
• The scorer name is teamai:routing:<skill> for a root skill and teamai:routing:<namespace>:<skill> for a skill from a skills namespace. A namespace cannot contain :, and a --scorer value with a / is read as a file, so : is the separator. Two namespaces' release are never counted together. In the rubric, <skill> is the bare name, matched against skills.available[].name with its namespace, and against skills.used.
• The names are generated and reserved: a scorer file name cannot contain :. This reuses the generated pin flag of #1180's correctness scorers, and adds resolving generated names for session entries and score --session --scorer.
• A session's routing questions always go in one judge call, whatever batch says. With batch: true, the team's session scorers that resolve to the same judge tool and model join that call, as batching does. With batch unset, the routing call takes no team scorer. On demand, the routing questions of one command share one call, without --batch, under a lease too. On demand, a routing name resolves whatever skill_routing says, and only for a skill in that session's skills.available; otherwise scorer_not_found.
• There is no samplingRate: every finished session the judge leases gets them. Routing launches count toward the same daily_limit as the team's session scorers, and sessions over it wait for a later day. With batch: true, a session costs one launch.
• The rubric ships with teamai. Its sourceHash covers the rubric with the skill's name filled in, not the description, so a rewritten description stays in the same measurement. A teamai release that changes the rubric changes the hash, as a team editing a scorer does.

Acceptance criteria

1. The session above produces the evidence.json above, under teamai score --session and under a background session lease.
2. A pull after the session's own pull that changes the description of release leaves the description in the evidence as it was.
3. A skill from a namespace the member was not delivered, or installed only for another tool, never appears in skills.available.
4. A description that holds a declared team secret reaches the evidence masked.
5. A session whose captures recorded no descriptions has the gap skills_unknown, no skills key and no routing question. A session whose checkout had no team skills has skills: { available: [], used: [] } and no gap.
6. skills.used equals the label's used.skills for the same read.
7. With sessions: true and skill_routing: true at the root, the session above gets exactly the two labels above, from one judge launch, and so does the score command above.
8. A member with the ops scorers namespace active and skill_routing: false in scorers/ops/scoring.yaml gets no routing question.
9. Two skills namespaces ops and web that each have a skill release give teamai:routing:ops:release and teamai:routing:web:release, never one name.
10. With batch: true and a team session scorer review-quality with no model, the session's routing questions and review-quality share one launch.
11. With skill_routing unset everywhere, no routing question is asked.
12. Trying scorers/ops/routing-release.md with teamai score --scorer on a session that needed release and loaded it gives a control of missed, with no warning.
13. With batch unset and ten skills, every finished session the judge leases gets its ten routing labels from one launch, and review-quality, when it samples the session, from another.

Definition of done

docs/guide/scorers.md and docs/guide/zh-CN/scorers.md, a "Skill routing" section under finished sessions, led by examples in this order: the scoring.yaml lines, the session's evidence.json, the score output, the dashboard counts with precision and recall worked out, the rubric, the namespace override, and the per-skill scorer with the batch and samplingRate note. Each example gets one or two sentences of prose, no more. The per-skill scorer is introduced by a `scorers/ops/routing-release.md`: line, so the scorer guide's example test parses it; the test creates the ops/ directory. skill-data/ gets the skill_routing line where it lists scoring settings.

• The tests in Test plan pass, including the e2e for criteria 1, 2, 5, 6, 7, 8, 12 and 13.
• The PR records one real-CLI run of a finished session with skill_routing: true and its labels.

Test plan

Unit tests for the capture's descriptions and namespaces, the manifest (masking, the tool filter, the gap, the empty case), and routing under root and namespace settings. E2E with a fixture session and a fake judge for criteria 1, 2, 5, 6, 7, 8, 12 and 13. One real-CLI run recorded in the PR.

Out of scope

• A setting per skill or per skills namespace, or a list of skills to leave out.
• Routing on workflow passes and benchmark trials.
• Skills that are not team skills: the member's own, plugins, and other tools'.
• Computing precision or recall in teamai (#1472 counts by value).
• List-valued answers (#1391).

Dependencies

• Blocked by: #1180. Its PR (#1379) adds the generated pin flag and the reserved-name rule, which routing questions reuse rather than adding a second kind: plan pins teamai:correctness:<task> the same way, and a workflow step's scorers now accept a name with :. The evidence part does not need it and can land first as its own PR.
• Touches: #1379 (also edits src/scorers/evidence.ts), #1269 (hand labels apply unchanged).
• Reading the results: #1472 counts labels by value in the scorer view of #1182, which precision and recall need.