[Bug]: Shared and nested bounded pools can starve queued child work

Start by reading the executor configuration and the root, child-context, step, and map/parallel coordinator execution paths, then inspect the nested2 LMI fixture and related work in #735 and #726. Reproduce the shared-pool and single-invocation nested starvation cases with explicit admission barriers, bounded timeouts, and cleanup. Done means supported configurations show deterministic progress and capacity behavior, with unsupported configurations diagnosed and lifecycle and replay guarantees covered.

Scope

Extracted from section 2 of #726: Shared fixed pools can starve their own steps; arbitrary nested bounded pools remain a progress problem. This issue tracks the remaining executor-progress and resource-capacity problem.

Open PR #735 provides an opt-in invocation-owned executor as a partial mitigation for contention between concurrent invocations. Shared executors configured with withExecutorService(...) retain their existing behavior, and an invocation-owned bounded pool can still starve on sufficiently nested synchronous work. Merging #735 must not close this broader issue.

Actual Behavior

Root handlers, steps, child contexts, and map/parallel coordinators can run on the same user executor. A parent that synchronously waits for its child keeps its worker occupied. If all workers are occupied by waiting parents, their queued children cannot start.

The original local reproduction on SDK v2.2.0 (638ded39318fbe8004fd74bc1c24da315fc96d29) shares a fixed pool of size 2 across two concurrent invocations. Both roots reach a barrier and then call a synchronous step. The pool has 2 active handler tasks and 2 queued steps, with neither invocation making progress. Increasing the pool to 4 releases that particular case.

Separating invocation capacity does not eliminate the same dependency cycle inside one invocation: a root and a child context can occupy a two-thread pool while the child waits for another queued step. Moving only roots also leaves child contexts and map/parallel coordinators able to exhaust the capacity needed by their descendants.

The default cached platform-thread pool avoids this particular fixed-capacity deadlock but can amplify thread and stack-memory usage. Limiting map/parallel maxConcurrency alone does not account for blocked roots, child contexts, or coordinators. LMI PerExecutionEnvironmentMaxConcurrency controls a different level of concurrency.

Steps to Reproduce

1. Configure withExecutorService(Executors.newFixedThreadPool(2)) and share the resulting DurableConfig across two concurrent DurableExecutor.execute() calls.
2. Synchronize both root handlers at a barrier, then have each call one synchronous step. Observe the two active parents and two queued steps, and verify that increasing the pool capacity releases the reproduction.
3. Add a separate nested control: keep a root and synchronous child handler active in the same bounded pool, then have the child submit and synchronously wait for further step/map/parallel work. Observe queued descendants with every worker occupied by a waiting ancestor. Run the equivalent control with an invocation-owned bounded executor when #735 is available.
4. Use explicit admission/barrier handshakes so setup delays cannot be mistaken for starvation. Provide bounded test timeouts and test-owned cleanup to release the intentionally blocked scenarios.

The complete original 2.x diagnostic class remains in #726 as historical evidence. Its assertions describe the old behavior; it is not a current 3.x acceptance test. The existing nested2 LMI fixture provides additional shared-pool nested diagnostics. #735 retains that fixture but excludes it from its 12 gating cloud cases, so a green #735 gate is not evidence that nested progress is fixed.

Expected Behavior

Define supported executor and nesting contracts that allow parent/child progress while controlling thread, queue, and memory growth. Diagnose detectable unsupported configurations and document limits that cannot be determined at configuration time.

Candidate approaches include separate execution capacity for blocking orchestration, avoiding blocking parents on the capacity needed by their children, or an appropriate virtual-thread/admission design while preserving the Java 17 baseline. A larger fixed pool or a virtual-thread choice alone is not a complete lifecycle/resource policy. This issue does not prescribe a particular implementation.

Acceptance Criteria

• Two concurrent invocations making synchronous step calls progress under each supported configuration; unsupported shared-pool configurations have explicit diagnostics/documentation.
• Deterministic root/child/step and nested map/parallel controls verify the supported nesting contract, including capacity exhaustion within a single invocation-owned bounded pool. Do not claim arbitrary nesting is solved by moving only roots or creating a fresh fixed pool.
• Capacity/admission accounting includes roots, child contexts, and blocking coordinators as well as step work; document and test thread/queue/resource growth under invocation concurrency and fan-out.
• Setup barriers, child admission, progress deadlines, and cleanup are independently observed so timeouts and pool-expansion escapes cannot be reported as successful progress.
• Preserve caller-owned executor lifetime, invocation isolation, owner/End ordering, deadline/cancellation semantics, replay identities, checkpoint ordering, and side-effect counts.
• Retain the shared/nested negative controls until addressed. When a configuration is claimed supported, promote the corresponding diagnostic to acceptance coverage and demonstrate overlapping same-JVM LMI invocations and nested progress. Do not count exclusion of nested2 from a gating suite as a fix.

Related Work

• #735: focused invocation-owned executor isolation; an opt-in mitigation for cross-invocation contention, with arbitrary nested starvation outside its scope.
• #726: hard cleanup/return bounds, LMI runtime-slot/worker recovery, and deployed lifecycle acceptance.
• #838 / #733: deadline admission and cooperative cancellation, including late descendant-admission safeguards; cancellation can stop an invocation but does not establish successful executor progress.
• #782 / #731: merged actual-task registry and owner/End ordering; #728 supplies the LMI harness.
• #12 and #297: earlier executor deadlock and pool-diagnostic discussions. The historical shared-pool reproduction here uses two concurrent invocations on v2.2.0.