tier-check reports 0/30 server conformance despite all tests passing

Open
#182 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
node.js, typescript
Domain
testing-qa, tooling

Research direction

Reproduce the issue with the conformance server and compare the tier-check command with the regular server command in @modelcontextprotocol/conformance v0.1.15. Trace how each command counts server scenarios, then verify that tier-check reports 30/30 (100%) when all 30 scenarios pass.

Written by the indexing model from the issue text.

Description

Summary

Running tier-check against a server that passes 30/30 conformance tests reports 0/30 (0%) server conformance.

Steps to Reproduce

# Start server (tower-mcp conformance server on port 3001)
cargo run --manifest-path examples/conformance-server/Cargo.toml &

# Regular conformance suite: 39/39 pass (30 scenarios, some with multiple checks)
npx @modelcontextprotocol/conformance@0.1.15 server --url http://127.0.0.1:3001/mcp
# Output: Total: 39 passed, 0 failed

# tier-check: reports 0/30
npx @modelcontextprotocol/conformance@0.1.15 tier-check --repo joshrotenberg/tower-mcp --conformance-server-url http://127.0.0.1:3001/mcp

Expected

tier-check should report 30/30 (100%) server conformance, matching the regular server command results.

Actual

Conformance:
                 2025-03-26 2025-06-18 2025-11-25              All*
✗ Server                  —       0/26       0/30        0/30 (0%)

Environment

  • @modelcontextprotocol/conformance v0.1.15
  • Server: tower-mcp conformance-server (passes all 30 scenarios / 39 checks)
  • macOS, Node 20
Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
4d 7h
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from modelcontextprotocol/conformance

All issues in modelcontextprotocol/conformance

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.