feat: add routing-focused eval support
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start by reviewing the current case schema and agent adapter interfaces to see how cases are modeled and whether triggered or loaded Skills are exposed. Compare the proposed routing_expect fields and determine what a skipped or unsupported result would require; done means the scope and adapter support are defined clearly enough to implement.
Written by the indexing model from the issue text.
Description
Motivation
skill-up is good at evaluating whether an Agent completes a task correctly once a Skill is installed. For Skill development, another important question is whether the Skill should be triggered for a prompt at all.
Routing behavior matters because:
- a Skill should trigger for intended prompts
- it should not trigger for unrelated prompts
- in multi-Skill environments, one Skill should not over-trigger or shadow another Skill
Proposed capability
A routing case could look like this:
id: route-code-review-positive
tag: routing
input:
prompt: "Review this pull request and identify risks"
routing_expect:
must_include: [code-review]
allow_extra: false
max_skills: 1
A negative case could look like this:
id: route-code-review-negative
tag: routing
input:
prompt: "Write a poem about winter"
routing_expect:
must_not_include: [code-review]
Possible semantics:
must_include: expected Skill names must be selected or loadedmust_not_include: listed Skills must not be selected or loadedallow_extra: false: no other Skills should be triggeredmax_skills: upper bound for triggered Skills
Why this matters
Without routing evals, a Skill can pass positive task cases but still be harmful in practice:
- it may trigger too often
- it may respond outside its intended scope
- it may cause negative delta in multi-Skill setups
Initial small scope
To keep the first version manageable, this could start as:
- model routing cases as normal cases with
tag: routing - add a
routing_expectschema - support only engines that expose triggered/loaded Skill information
- return a clear skipped/unsupported result when routing info is unavailable
Open questions
- Do current agent adapters expose enough information to reliably identify which Skills were triggered?
- Should routing be a top-level suite type, or regular cases with
tag: routingplusrouting_expect? - Should the first version support only single-Skill routing, leaving multi-Skill routing for later?
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day