WARNING severity is inconsistent between client and server runners (client warnings can block Tier 1, server warnings cannot)

Open
#430 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
testing-qa

Research direction

Read src/runner/client.ts, src/runner/server.ts, src/expected-failures.ts, and src/tier-check/tier-logic.ts to compare how WARNING affects runs, baselines, and tier rates. Run the relevant conformance and tier-check tests, then confirm which warning semantic is chosen and verify that all three paths apply it consistently without changing current Tier 1 results.

Written by the indexing model from the issue text.

Description

Problem

The two runners treat WARNING checks differently:

  • Client runner (src/runner/client.ts): overallFailure = failed > 0 || warnings > 0 || ... — any WARNING fails the scenario run.
  • Server runner (src/runner/server.ts): the pass/fail denominator counts only SUCCESS and FAILURE; warnings are reported but never affect the result.

Since tier-check requires pass_rate = 1.0 for Tier 1 on both conformance and client_conformance (src/tier-check/tier-logic.ts), the same severity mapping (SHOULD → WARNING) is a Tier 1 blocker on the client side but cosmetic on the server side. Whether an SDK's tier is affected by a SHOULD-violation currently depends on which side of the wire the scenario tests, which is hard to defend in a tiering dispute.

Related: src/expected-failures.ts counts WARNING as failure in baseline mode, which is a third distinct behavior.

Ask

Pick one semantic and apply it in both runners (and baseline mode). Options:

  1. Warnings never fail runs or tiers anywhere (server behavior today); SHOULD compliance becomes report-only.
  2. Warnings fail everywhere (client behavior today); Tier 1 then requires SHOULD compliance on both sides.
  3. Keep warnings non-failing for the run but surface a separate SHOULD-compliance rate in tier-check so tiering can reference it explicitly.

Context: noticed while landing #322, whose DELETE-status check is a WARNING because the spec pins no success status (#429). No current Tier 1 SDK emits warnings on either side, so any of the above can land without demoting anyone today.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
4d 7h
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from modelcontextprotocol/conformance

All issues in modelcontextprotocol/conformance

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.