[Durable Execution] Complete LMI deadline handling, teardown, and integration coverage

This is a multi-part tracking issue with no single change to make. Start by reading WorkflowCancellation.cs, CheckpointBatcher.cs (DrainAsync and DisposeAsync), DurableExecutionHandler.cs (RunAsync) and the WorkflowCancellationTests.cs race-related tests. Done means the listed acceptance tests pass locally and on real LMI, which needs maintainer design decisions first.

Describe the feature

Complete Lambda Managed Instances (LMI) support for Amazon.Lambda.DurableExecution by handling invocation deadlines, making suspension cleanup ordering explicit, and adding durable-specific LMI tests.

LMI runs concurrent invocations in one process and does not forcibly stop code when an invocation times out. A timed-out invocation can therefore retain a runtime worker and continue side effects while healthy peers run.

This is a source-review tracking issue, not a report of a reproduced cloud failure.

Current behavior and evidence

The basic concurrency architecture is already present: WrapAsyncCore creates execution state, operation IDs, termination/cancellation state, a checkpoint batcher, and context per invocation. Logging scopes use AsyncLocal, and RuntimeSupport already dispatches concurrent requests.

The remaining gaps are:

1. No invocation deadline is connected to durable cancellation. State hydration omits the supported cancellation token, and WorkflowCancellation responds only to TerminationManager. ILambdaContext.RemainingTime is not used to stop workflow work, hydration, checkpoint requests, or retries.
2. Checkpoint cleanup has no invocation deadline. DrainAsync and DisposeAsync await their worker without a deadline. They do join it before a completed wrapper return; the problem is that a slow/stalled worker can keep the invocation open beyond its deadline.
3. Suspension cancellation/disposal ordering needs a deterministic regression. RunAsync returns when termination wins without joining the losing workflow task. Cancellation runs through an asynchronous continuation, and disposal does not wait for its delivery. The possible dispose-before-cancel ordering can skip cancellation because the continuation catches ObjectDisposedException. Existing tests acknowledge asynchronous delivery; the full wrapper race has not been reproduced in this audit.

Use Case

Run durable workflows concurrently on LMI while one execution waits, retries, fails, or times out, without leaking invocation state, exhausting worker capacity, or allowing stale SDK work to interfere with a healthy execution.

Proposed Solution / work items

• Derive an invocation-scoped deadline from RemainingTime; propagate cancellation through hydration/pagination, backend requests and retries, and SDK-owned work. Stop admission of new work once teardown starts.
• Define and implement a bounded cleanup budget. Ensure cancellation delivery completes before disposing cancellation resources; observe task failures and finish or safely contain SDK-owned work.
• Preserve checkpoint ordering, token ownership, persisted successes/failures, and replay semantics. An invocation deadline must not accidentally become a permanent durable-execution failure; define handling of already-admitted/in-flight writes.
• Preserve intentional durable suspension. Do not simply await the losing workflow forever: waits deliberately leave pending tasks. Define cleanup of active work separately and document cooperative cancellation requirements and limits for arbitrary user code.
• Verify invocation-local request IDs, log scopes, trace parents, serializer usage, and shared-client behavior. Document thread-safety requirements for shared handler fields, custom serializers, clients, and temporary files.
• Document and validate supported LMI runtime/deployment settings and resolved package versions. The official .NET guide requires Core >=2.7.1, RuntimeSupport >=1.14.1 when packaged, and AWSSDK.Core >=4.0.0.32. Current Core/RuntimeSupport already exceed the minimums. Document applicable optional instrumentation versions/restrictions and use context.TraceId rather than _X_AMZN_TRACE_ID.
• Add a deployable LMI durable example and CI coverage for the documented setup.

Required test coverage and acceptance criteria

• Concurrent independent executions: run two or more invocations through the same real bootstrap/handler instance. Verify isolated execution ARNs, operation identities, checkpoint tokens, results/errors, serialization contexts, request IDs, logging scopes, and trace parents.
• Replay while a peer is live: cover steps, waits, retries, callbacks/invokes, child contexts, and map/parallel operations across initial invocation and resume. Completed user bodies must be skipped and checkpointed success/failure reproduced; another execution's logs or state must remain unaffected.
• Teardown ordering: deterministic tests for success, failure, suspension, and deadline paths, including cancellation-before-disposal, cancellation delivery under load, late completion/failure, and cooperative versus non-cooperative work. Assert actual lifecycle boundaries using barriers/signals instead of sleeps.
• Slow/failing backend: stall hydration, pagination, checkpoint writes and retries; exercise transient failures and missing/stale checkpoint tokens. Verify deadline handling, bounded cleanup, rejection of newly admitted work after teardown, and the documented outcome of in-flight requests.
• Real LMI timeout isolation: keep a healthy peer active while another invocation times out. Prove the original execution environment remains usable and admits new work after cleanup; replacement of the environment or a later successful retry must not substitute for recovery evidence.
• Real LMI suspend/resume: use a capacity-provider-backed published function with supported runtime/architecture combinations, concurrency 1 as a baseline and concurrency >=2 for overlap. Prove simultaneous invocations occur in the same process/environment, then validate histories and side-effect counts across replay.
• Repeated warm execution: repeat success, failure, suspension and timeout cases; verify tasks, resources and retained invocation state do not grow without bound.
• Keep ordinary Lambda behavior and current component/conformance tests passing. Record LMI configuration, SDK revision, histories, lifecycle markers and test results. Coordinate any language-neutral requirements/runner changes with the conformance repository.

Existing RuntimeSupport concurrency tests and durable integration deployment do not establish this combined LMI contract. Completion requires recorded local regressions and real LMI results.

AWS .NET SDK and/or Package version reviewed

Repository 3ff8ae1ba913ffc1113b49531016f6d1c019bb78: DurableExecution 2.2.0, Core 3.4.0, RuntimeSupport 2.3.0, AWSSDK.Lambda 4.0.13.1.

Targeted .NET Platform

.NET 8 and .NET 10 where supported by LMI.

Operating System and version

AWS Lambda Managed Instances Linux environments. This issue is based on source inspection; no live LMI test was run for this report.

Related work

Rust LMI compatibility tracker covers the corresponding runtime integration and cross-invocation validation work.