Standardize how scenarios report setup/execution failures

Open
#248 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Quiet
Tech stack
typescript
Domain
testing

Research direction

Start by reading the scenario implementations and the proposed client-helper.ts utility, then compare how setup and execution failures are currently reported. Decide on the setup-failure convention described in the issue and verify that scenarios report failures consistently while preserving --suite, --scenario, and expected-failures behavior.

Written by the indexing model from the issue text.

Description

A scenario that can't execute (connect failure, fixture missing, capability not advertised) should count as a FAILURE. The escape hatches are scenario filtering (--suite/--scenario) and the expected-failures baseline (#99) — not in-scenario skipping or silent passes.

Today every scenario hand-rolls this: most wrap the body in one big try/catch and emit their primary check ID as FAILURE on any error. That's fine for single-check scenarios but gets awkward with multiple checks at different severities (#200 discussion) — which check ID gets the connect failure is arbitrary, and the error message ends up under a label that has nothing to do with the actual problem.

Options:

  • Emit all of the scenario's check IDs as FAILURE with the setup error message (consistent output shape)
  • Emit a dedicated <scenario-name>-setup check (semantically cleaner, but adds non-spec-mapped IDs)
  • A client-helper.ts utility that scenarios call on setup failure, so the convention is enforced in one place

The helper is probably the best and then we can evolve the behavior from there. Having all the check-id's emitted requires a scenario knowing all its checks, which might be useful but is a larger change. The -setup check is probably reasonable as a first cut for this.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
4d 7h
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from modelcontextprotocol/conformance

All issues in modelcontextprotocol/conformance

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.