feat(mcp): make mocked MCP servers closer to the real server
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go
- Domain
- testing-qa
Research direction
Start with the mock server in internal/mcp/mock_server.go, where tools/list builds the generic tool descriptions and tool_responses are returned, then read internal/mcp/provisioner.go where config_ref is loaded. The issue proposes a SUP covering a tools map, per-call response selection, error results and a call log, so agree that shape with maintainers before coding. Done means a maintainer-approved SUP, or the small PRs for items 1 and 3 that the author offered.
Written by the indexing model from the issue text.
Description
Problem / motivation
Mocked MCP servers are why I picked skill-up over other eval runners, and fixture-based mocks are still quite thin next to a real server. I'd like to work on this, and want to agree the shape with you first.
What I see on current main for tool_responses mocks
- Tools look generic.
tools/listadvertisesMocked MCP tool <name>with an any-object schema (mock_server.go#L131). The agent never sees the real descriptions or parameters, which may change its tool choice compared with production. - One response template per tool. Arguments can be interpolated into it, but a fixture can't pick a different response by argument values or call order (mock_server.go#L148), so distinct page contents or a failure-then-success retry can't be modelled.
- Declared tools can't fail on purpose. A fixture can't make a declared tool return a protocol error or a result with
isError: true, so a Skill's error handling is hard to test. Successful replies are also always one text block (mock_server.go#L143), so a nativestructuredContentresult can't be mocked either. - No record of calls, and no trace of the fixture. The mock keeps no log of what it received, so tool-call assertions depend on each engine's transcript parsing. Separately, the report doesn't show which fixture was in effect (N1 in SUP-0003).
validatenever reads the fixture. Theconfig_reffile is first loaded when the server is provisioned (provisioner.go#L122), so a missing file or a YAML or type error in it shows up only once a run starts.- Custom engines get no automatic MCP installation.
CustomAgent.InstallMCPlogs the servers and returns (custom.go#L114).
Proposed solution
- 1 to 4 in one SUP, since they add fixture fields and touch the report. For 1, an optional top-level
toolsmap withdescriptionandinput_schema, kept apart fromtool_responsesso existing fixtures keep their meaning, and today's behaviour when it is absent. The schema is shown to the agent only; call arguments are not checked against it. - For 4, one question first. Should a call log serve debugging, judges, or both? That decides whether it touches judge inputs.
- 5 as a direct PR, limited to what loading already catches (missing file, YAML or type errors). Any new rejection rules would go in the SUP.
- 6 outside this effort. It touches the custom engine contract.
If you'd rather take 1 and 3 as small PRs without a SUP, that works for me too. I have a draft for 1 ready either way.
Alternatives considered
A user can write their own stdio server and declare it as mode: real. That covers stateful cases, but each user then implements mock behaviour and per-case fixture selection themselves.
Additional context
#339 (merged) fixed the tool_called examples in the docs to use the mcp__<server>__<tool> names engines record, and documents how tool_responses mocks behave today.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 134
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 61
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from alibaba/skill-up
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remotePossibly taken @bababoom721 claimed this 1 day ago. Open
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 30/100
alibaba/skill-up#408 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
alibaba/skill-up#407 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
alibaba/skill-up#406 · 1 comment ·
Maintainers usually reply within 1 day
All issues in alibaba/skill-up
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
OneBusAway/maglev#2145 ·
Maintainers usually reply within 2 days
-
priority/medium status/ready type/chore
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
discobox-ai/discobox#1357 ·
Maintainers usually reply within 1 day
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itPossibly taken @fzn0x claimed this today. Openbug priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
edit_file diff rendering panics on invalid UTF-8 textPossibly taken @TDnorthgarden claimed this today. Openarea/tui
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
docker/docker-agent#4955 ·
Maintainers usually reply within 1 day
-
enhancement ui/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
jbigler/railswitch#243 ·
Maintainers usually reply within 1 day