feat: add routing-focused eval support

Open
#166 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
go
Domain
ai, testing

Research direction

Start by reviewing the current case schema and agent adapter interfaces to see how cases are modeled and whether triggered or loaded Skills are exposed. Compare the proposed routing_expect fields and determine what a skipped or unsupported result would require; done means the scope and adapter support are defined clearly enough to implement.

Written by the indexing model from the issue text.

Description

Motivation

skill-up is good at evaluating whether an Agent completes a task correctly once a Skill is installed. For Skill development, another important question is whether the Skill should be triggered for a prompt at all.

Routing behavior matters because:

  • a Skill should trigger for intended prompts
  • it should not trigger for unrelated prompts
  • in multi-Skill environments, one Skill should not over-trigger or shadow another Skill

Proposed capability

A routing case could look like this:

id: route-code-review-positive
tag: routing

input:
  prompt: "Review this pull request and identify risks"

routing_expect:
  must_include: [code-review]
  allow_extra: false
  max_skills: 1

A negative case could look like this:

id: route-code-review-negative
tag: routing

input:
  prompt: "Write a poem about winter"

routing_expect:
  must_not_include: [code-review]

Possible semantics:

  • must_include: expected Skill names must be selected or loaded
  • must_not_include: listed Skills must not be selected or loaded
  • allow_extra: false: no other Skills should be triggered
  • max_skills: upper bound for triggered Skills

Why this matters

Without routing evals, a Skill can pass positive task cases but still be harmful in practice:

  • it may trigger too often
  • it may respond outside its intended scope
  • it may cause negative delta in multi-Skill setups

Initial small scope

To keep the first version manageable, this could start as:

  1. model routing cases as normal cases with tag: routing
  2. add a routing_expect schema
  3. support only engines that expose triggered/loaded Skill information
  4. return a clear skipped/unsupported result when routing info is unavailable

Open questions

  1. Do current agent adapters expose enough information to reliably identify which Skills were triggered?
  2. Should routing be a top-level suite type, or regular cases with tag: routing plus routing_expect?
  3. Should the first version support only single-Skill routing, leaving multi-Skill routing for later?
Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.