Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Expose expected-vs-emitted check coverage per scenario

Open
#451 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
typescript
Domain
cli, testing-qa

Research direction

Trace how Scenario and ClientScenario define or execute checks, then follow result construction into terminal summaries and machine-readable output. Review the setup-failure and passing thin-scenario cases first. Done means eligible expected and emitted coverage is reported consistently, applicability gates do not create false gaps, and both cases have tests.

Written by the indexing model from the issue text.

Description

A scenario can emit only a thin subset of its checks while the reported denominator makes that subset look complete.

The concrete case from mcpkit: auth/authorization-server-migration reported 2 passed, 1 failed for months. The initial wire handshake failed, so roughly 76 other checks never emitted. PR #327 makes the setup failure explicit, but the result still cannot show how much of the scenario did not run.

The harder case has no setup failure: setup succeeds, only accept-path checkpoints emit, and the thin scenario result passes.

Add a per-scenario expected-vs-emitted coverage signal to the upstream results. It should be visible in both terminal summaries and machine-readable output. A count is enough to expose the gap; stable IDs would also identify the missing checks.

Today Scenario and ClientScenario do not declare their expected check IDs, and some checks are dynamic or version-gated. The implementation will need a stable expected set or another baseline that excludes checks which are not applicable to the selected spec version or extension.

Acceptance criteria:

  • Every executed scenario reports expected and emitted check counts, or equivalent coverage data.
  • Missing eligible checks remain visible when every emitted check passes.
  • Machine-readable results expose the coverage signal for downstream consumers.
  • Spec-version, extension, and documented applicability gates do not create false gaps.
  • Tests cover both a setup failure and a passing thin-shadow result.
  • SDKs do not need to maintain their own per-scenario count snapshots.

@panyam offered to implement this in #327.

Prior art: https://github.com/panyam/mcpkit/pull/1115

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
4d 7h
Merged PRs (30d)
6

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from modelcontextprotocol/conformance

All issues in modelcontextprotocol/conformance

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.