Provide a correlated cancel-and-drain contract for reusable JSON-RPC test hosts

Start with MtpJsonRpcConnection.CancelPendingRequest in src/Platform/Microsoft.Testing.Platform.ServerMode.Client.Sources/Client/, which completes the pending request and sends the cancel notification without awaiting it, and with SendRequestAsync, which removes the pending entry in finally. Then read IMtpServerClient.cs and MtpServerClient.RunCoreAsync to see where the run GUID is generated. The work is a design proposal covering a stop/drain contract, run identity, and finality negotiation, so 'done' means an agreed API and capability spec plus the acceptance scenarios listed in the issue.

Problem

A server-mode consumer cannot currently request cancellation and await server quiescence through the supported source client's high-level API. This matters for Stryker.NET: a persistent host reads a shared active-mutant ID. Switching that ID after a locally cancelled RunTestsAsync task can overlap the previous run's test/cleanup code and attribute late results to the next mutant.

This is a lifecycle/API gap, not a request for MTP to decide which test kills a mutant, cache discovery, or implement mutation scheduling.

Versions and source evidence

Checked on 2026-10-09:

• Latest stable Microsoft.Testing.Platform.ServerMode.Client.Sources: 2.5.1 (published 2026-10-07), consumed by current Stryker.NET master.
• MSTest 4.5.1, corresponding MTP 2.5.1.
• The same cancellation sequence remains in TestFx main at c5e1f5c613fa02ffa507db333e4702da6c97e488.

MtpJsonRpcConnection.CancelPendingRequest calls pending.Completion.TrySetCanceled(cancellationToken) and then fire-and-forgets SendCancelNotificationAsync(id). SendRequestAsync removes the pending entry in finally. Therefore OperationCanceledException proves neither that the cancel notification was sent nor that the correlated terminal response, test cleanup, and update drain have completed.

IMtpServerClient documents normal terminal-response/update ordering, but exposes no separate stop-request/drain operation. MtpServerClient.RunCoreAsync also generates the run GUID internally, while update events expose RunId; consumers cannot establish the expected identity before the first update.

Minimal reproduction

1. Start a persistent MSTest MTP application using the official source client.
2. Run a test that waits using TestContext.CancellationToken, then performs observable asynchronous cleanup in finally. Also exercise a test that ignores cancellation.
3. Wait for its in-progress notification and cancel the token passed to RunTestsAsync.
4. Observe that the request task can complete with OperationCanceledException before the cleanup marker is written or the server's correlated terminal response is received.
5. Reuse the host immediately or replace an external run-control value. Previous work can still be active. Detaching the old update handler and attaching a new one is not a drain barrier.

The timing in step 4 follows directly from the client implementation; it is not evidence that a particular server/provider has finished. A consumer can test actual terminal drain by retaining response correlation while sending the existing cancel notification, but reaching below the supported high-level API is not an appropriate production solution.

Proposed contract

Prefer a small additive high-level client API, with a server capability only where stronger server semantics are needed. The exact name is open for discussion. One possible shape is a run request/options object with a caller-supplied RunId, an execution-stop token, and a separate wait-cancellation token, returning a completion outcome.

Ownership and identity

• MTP owns provider execution, per-request teardown, result transport ordering, and the terminal response. The client owns response correlation and synchronous notification dispatch. The consumer owns its asynchronous notification work and must await that too.
• Allow the caller to provide a run GUID (or obtain an operation handle before execution starts), so discovery/run updates and completion can be matched from the first notification. Reject duplicate active identities.
• Cancellation targets the exact JSON-RPC request; it must not cancel unrelated runs, discovery requests, or the whole reusable host.

Request stop versus cancel a wait

• Requesting execution stop sends a correlated cancellation notification without removing/completing the pending response entry. Continue reading and dispatching that run's updates until its terminal response is processed.
• Return a completion outcome distinguishing normal completion from stopped execution; retain real partial results and artifacts. A race where the run finishes before stop arrives is normal completion, not a false failure.
• Cancelling the caller's wait is a different operation. It may return promptly, but must explicitly make no quiescence/reuse promise. Expose the undrained state or invalidate that connection for further runs.
• Deliberate bail, timeout, and user cancellation are consumer stop reasons. Keep mutation classification and baseline-failure/retry policy in Stryker; MTP need only report completion/stopped/aborted status and reusable versus unknown lifecycle state.

Terminal and drain guarantees

• Stop scheduling new tests, signal running tests using the provider's supported graceful-stop/cancellation capability, await test/framework cleanup and per-request end hooks, then drain accepted result messages through consumers and transport before emitting the terminal response.
• Keep the drain/cleanup path independent of the execution-stop token. A cancelled consumer loop must not silently discard already accepted results or overlap its own disposal.
• No notification for that request may follow its terminal response. Client completion means all synchronous update handlers for that run have returned, not that arbitrary consumer-dispatched asynchronous work has completed.
• Do not fabricate outcomes for tests that did not complete. Retry attempts marked retry.is-superseded must remain identifiable, and incomplete retry sequences must not become final failures.

Bounds, concurrency, and fallback

• Consumers set a bounded drain deadline. A test or cleanup handler that does not cooperate must not be treated as quiescent: terminate/discard the process and verify its exit before starting another run with changed external state.
• Retain the pending entry until terminal response, connection failure, or explicit bounded abort. Observe late send/read failures; avoid fire-and-forget success-shaped cancellation.
• Define reentrancy: update handlers can request stop without blocking the ordered read loop; they cannot synchronously wait for completion from that loop. A one-run connection cannot release its gate merely because the waiting caller cancelled.
• Multi-request servers need independent operation state. A consumer running several assemblies must drain/discard every started host before publishing its next shared control state; unstarted assemblies can simply be skipped.

Compatibility and shipping

• Preserve existing token semantics for existing callers. Additive stop-and-drain behavior must not silently change the meaning of their cancellation token.
• Negotiate an explicit cancellation-drain capability if server cleanup/ordering guarantees differ from older implementations. Package version or the first terminal-looking test update is not a substitute for that capability.
• For an older/unknown server, report unsupported or complete with not reusable; do not infer safety from a locally cancelled task. The safe consumer fallback is bounded discard/restart.
• Cover the shipping source-client package and aligned server/provider packages, and assess rel/4.5 servicing before downstream consumers depend on host reuse after cancellation. A new client alone cannot create missing server guarantees.

Acceptance scenarios

Use the real source client and packaged MSTest MTP hosts: normal completion; stop after a final failure; stop racing normal completion; cooperative cancellation with delayed cleanup; noncooperative test/cleanup and bounded abort; independent wait cancellation; transport failure/crash; zero results; superseded retry failure followed by pass; late/wrong-run updates; repeated runs and multiple assemblies; and an older server without the capability. Assert exact run IDs, terminal response ordering, cleanup completion, partial results, process exit, and absence of updates leaking into the next run.

Related work and consumer mitigation

• #8580 proposes application-level phased shutdown. This request is specifically the reusable per-request JSON-RPC source-client stop/drain contract.
• #10888 proves cancellation reaches a running MSTest test; its description explicitly distinguishes that from final-result reporting and does not establish reusable-host quiescence.
• #10357 addresses message-bus cancellation/drain sequencing. Verify that guarantee applies to the per-request server path and the releases consumers will ship with.
• stryker-mutator/stryker-net#3674 is an older open remote-bail proposal; it is not evidence of supported safe reuse.

Stryker's conservative mitigation is to retain actual partial outcomes and discard cancelled hosts before switching mutants. Supported cancel-and-drain would allow cooperative cancellation to retain a warm host without relying on internal transport manipulation or settle delays.

Design clarification: streamed result finality

Result finality is a separate capability from cancellation drain. Independent
design reviews identified that a reusable stop contract alone is insufficient
for a consumer to stop after a streamed failure.

In shipped MSTest 4.5.1, MSTestTestNodeConverter adds RetryAttemptProperty
only when RetryAttemptNumber > 1 or IsSupersededRetryAttempt is true.
MTP 2.5.1 serializes that property as retry.attempt and
retry.is-superseded. An ordinary terminal outcome therefore need not carry
either field. A client serving arbitrary or older providers cannot infer that
an unannotated failure is final merely because it has not observed a retry yet.
The first superseded failure could be the first evidence of retry participation.

Please define a positive, negotiated streamed-outcome-finality contract. Two
possible minimal representations are:

• A capability stating that every published terminal outcome is final unless
  it is explicitly annotated as superseded, with all retry producers required
  to obey that contract before advertising it.
• An explicit finality field on every actionable terminal outcome, not just
  on tests whose retries actually occurred.

The precise field/capability name is not prescribed. Advertised semantics must
also cover custom frameworks and adapters, dynamically generated child cases,
retry-attempt ordering, and cancellation of an incomplete retry sequence.
Test-attempt finality and whole-request quiescence remain different guarantees:
a final failure can justify consumer policy without implying that the rest of
the request or its cleanup has finished.

For older or unknown servers/providers, Stryker's conservative behavior is to
wait for the assembly's correlated terminal response before interpreting
unannotated results; only explicitly final retry outcomes qualify for
in-assembly cancellation. This intentionally does not promise streaming bail
for the default, unannotated single-assembly suite. Inferring finality from
absence of previously observed retries would be unsafe.

Verified packaged reproduction and consumer acceptance

The gap is now reproduced with the official 2.5.1 source client and an
unmodified MSTest 4.5.1 / MTP 2.5.1 application, not only by source inspection.

The Stryker fixture prepared in the consumer worktree is under
integrationtest/TargetProjects/MicrosoftTestPlatform/Cancellation.
It is not yet published or merged; no consumer PR was requested. The command
below is reproducible with that consumer patch. The minimal MSTest reproduction
steps above remain independent of publishing the Stryker mitigation.
SourceClient_CancelledWait_DoesNotProveServerCleanupHasFinished starts a real
host, waits for a marker written inside the executing test, cancels the ordinary
client token, and observes OperationCanceledException. It then waits for the
test's cleanup-start marker and asserts that the host is still alive and
cleanup has not finished. Cleanup is deliberately held behind a release
marker, making this a deterministic lifecycle proof rather than a settle-delay
test. The acceptance test finally force-discards and verifies exit of that host.
No private transport manipulation or direct cancel-frame injection is used.

The same validation runs seven real-host acceptance tests, including public MTP
pool caller cancellation followed by another mutant, cooperative and
noncooperative cancellation, timeout, final retry outcomes across host reuse,
and zero-result classification. All seven executed and passed without skips.

Seven real Release Stryker commands also completed with exit code 0 and the
expected generated JSON classifications:

Scenario	First mutant	Next mutants
Explicit-final cooperative bail	Killed	Survived, Survived
Disable bail	Killed	Survived, Survived
Noncooperative selected work	Killed	Survived, Survived
Timeout	Timeout	Survived, Survived
Host exit/crash	RuntimeError	Survived, Survived
Unannotated/legacy finality fallback	Killed	Survived, Survived
Initially failing test	Killed	Survived, Survived

Reproduction command in the Stryker checkout, after restoring the existing
CLI/unit projects' locked dependencies:

.\integrationtest\TargetProjects\MicrosoftTestPlatform\Cancellation\run-validation.ps1 -EvidencePath .\artifacts\mtp-cancellation

The script builds Release binaries, verifies that all real-host tests actually
ran, checks parent exit codes and report classifications, and saves TRX, logs,
per-process execution/cleanup markers, and outcomes.json. The consumer's
discard fallback is a mitigation; these successes do not establish that the
upstream API has gained a reusable cancellation-drain guarantee.