feat(mcp): make mocked MCP servers closer to the real server

Open
#357 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
testing-qa

Research direction

Start with the mock server in internal/mcp/mock_server.go, where tools/list builds the generic tool descriptions and tool_responses are returned, then read internal/mcp/provisioner.go where config_ref is loaded. The issue proposes a SUP covering a tools map, per-call response selection, error results and a call log, so agree that shape with maintainers before coding. Done means a maintainer-approved SUP, or the small PRs for items 1 and 3 that the author offered.

Written by the indexing model from the issue text.

Description

Problem / motivation

Mocked MCP servers are why I picked skill-up over other eval runners, and fixture-based mocks are still quite thin next to a real server. I'd like to work on this, and want to agree the shape with you first.

What I see on current main for tool_responses mocks

  1. Tools look generic. tools/list advertises Mocked MCP tool <name> with an any-object schema (mock_server.go#L131). The agent never sees the real descriptions or parameters, which may change its tool choice compared with production.
  2. One response template per tool. Arguments can be interpolated into it, but a fixture can't pick a different response by argument values or call order (mock_server.go#L148), so distinct page contents or a failure-then-success retry can't be modelled.
  3. Declared tools can't fail on purpose. A fixture can't make a declared tool return a protocol error or a result with isError: true, so a Skill's error handling is hard to test. Successful replies are also always one text block (mock_server.go#L143), so a native structuredContent result can't be mocked either.
  4. No record of calls, and no trace of the fixture. The mock keeps no log of what it received, so tool-call assertions depend on each engine's transcript parsing. Separately, the report doesn't show which fixture was in effect (N1 in SUP-0003).
  5. validate never reads the fixture. The config_ref file is first loaded when the server is provisioned (provisioner.go#L122), so a missing file or a YAML or type error in it shows up only once a run starts.
  6. Custom engines get no automatic MCP installation. CustomAgent.InstallMCP logs the servers and returns (custom.go#L114).

Proposed solution

  • 1 to 4 in one SUP, since they add fixture fields and touch the report. For 1, an optional top-level tools map with description and input_schema, kept apart from tool_responses so existing fixtures keep their meaning, and today's behaviour when it is absent. The schema is shown to the agent only; call arguments are not checked against it.
  • For 4, one question first. Should a call log serve debugging, judges, or both? That decides whether it touches judge inputs.
  • 5 as a direct PR, limited to what loading already catches (missing file, YAML or type errors). Any new rejection rules would go in the SUP.
  • 6 outside this effort. It touches the custom engine contract.

If you'd rather take 1 and 3 as small PRs without a SUP, that works for me too. I have a draft for 1 ready either way.

Alternatives considered

A user can write their own stdio server and declare it as mode: real. That covers stateful cases, but each user then implements mock behaviour and per-case fixture selection themselves.

Additional context

#339 (merged) fixed the tool_called examples in the docs to use the mcp__<server>__<tool> names engines record, and documents how tool_responses mocks behave today.

Dominant language
Go
Stars
1.5k
Forks
134
Avg merge
1d 22h
Merged PRs (30d)
61

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from alibaba/skill-up

All issues in alibaba/skill-up

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.