refactor(gateway): establish structured ownership for asynchronous tasks

Start at run_server and the shutdown sequence in crates/openshell-server/src/lib.rs, then trace the affected components listed in the issue, especially mutation_replay.rs and supervisor_session.rs. Establish the ownership and shutdown phases, add the required deterministic regression coverage for admitted lifecycle work, preserve the referenced guarantees, and update the named lifecycle documentation and diagnostics.

User Story

As an OpenShell operator, I want graceful gateway shutdown to account for all work the gateway has admitted, so that restart and local-compute cleanup do not depend on which detached Tokio tasks happen to finish before the process exits.

Problem Statement

The gateway starts asynchronous work through several unrelated lifetime mechanisms: detached connection tasks, detached request workers, recurring background loops, shutdown-aware workers whose handles are discarded, locally joined task groups, and the supervisor-session lifetime barrier added by #3547. There is no gateway-wide ownership model that closes admission, cancels work where appropriate, and joins work in the dependency order required by graceful shutdown.

This is a generalization of two concrete shutdown races. Closed issue #2826 identified accepted lifecycle mutations that can commit after the local-compute shutdown inventory is captured. Issue #3546 and PR #3547 demonstrate a second manifestation: a detached supervisor-session task can leave persisted ownership behind when the process exits. The local barrier in #3547 is an appropriate focused repair, but additional one-off barriers would make lifecycle reasoning increasingly fragmented.

Related open issue #3154 addresses the same design principle for Linux supervisor child processes: spawn, registration, waiting, and cleanup should be owned structurally rather than coordinated by caller convention. It is a separate subsystem and not a duplicate, but its deterministic interleaving-test requirement is a useful precedent.

Impact / Why This Matters

The current model makes graceful-shutdown guarantees depend on task scheduling. Accepted mutations can outlive the cleanup phase that should account for them, persistence cleanup can be interrupted, long-lived connections can continue carrying new HTTP/2 streams after the accept loop stops, and forever-running workers are cancelled only when the Tokio runtime is dropped.

Operators cannot reliably quiesce the gateway from outside because stopping the listener does not establish an application-level admission boundary for already accepted connections. The resulting failures can leave local compute running while the gateway is unavailable, block replacement supervisors with stale ownership, or abandon durable mutation admissions without a recorded result.

Acceptance Criteria

• Define a gateway task-ownership model that distinguishes request/connection work, state-critical owned mutations, supervisor sessions and relays, and recurring background workers.
• Define explicit shutdown phases and ordering constraints, including admission closure, lifecycle-mutation drain before the local-compute inventory, compute cleanup while supervisor reporting remains available, supervisor ownership cleanup, and final worker drain.
• Ensure tasks that can outlive their caller are registered before they can publish durable state, without an admission-versus-shutdown race.
• Ensure already accepted HTTP/2 connections cannot admit new state-changing work after gateway shutdown begins.
• Give every production background task an explicit owner and termination policy: graceful completion, cooperative cancellation, or abort with a documented reason.
• Bound each shutdown phase, aggregate actionable errors, and preserve safety when a task or persistence operation stalls.
• Replace the supervisor-session RwLock<()> lifetime barrier from #3547 once the shared model provides equivalent admission and cleanup guarantees.
• Incorporate the lifecycle-mutation race from #2826 with a deterministic regression test proving shutdown cannot miss eligible local compute created by an admitted request.
• Preserve the #3546 regression guarantees: owner cleanup completes before exit, stale sessions cannot delete replacement ownership, and VM restart recovery succeeds.
• Preserve Kubernetes workload continuity and existing Docker, Podman, VM, and managed-driver shutdown semantics.
• Document the final lifecycle and task-ownership invariants in architecture/gateway.md and update operator-facing shutdown diagnostics.

Technical Context

run_server owns the process-level shutdown sequence, but only retains and joins the main listener task. The listener stops accepting sockets after a watch signal, while each accepted TLS or plaintext connection is moved into an untracked tokio::spawn. Hyper serves many streams per connection, so awaiting the accept loop does not mean active connections or their RPCs have quiesced.

Several handlers then create further tasks. Durable mutation replay deliberately spawns before its first database write so caller cancellation cannot interrupt ownership, but the gateway does not retain that task for process shutdown. Supervisor connection setup, session loops, relay pumps, service-routing connections, WebSocket bridges, and response-stream producers use a mixture of detached tasks and local handles. Recurring workers likewise vary: compute loops and TLS reload receive the global shutdown watch, whereas credential refresh, SSH-session reaping, relay reaping, health polling, extension-token rotation, health serving, and metrics serving rely on runtime teardown or channel closure.

The code already contains useful local patterns. The compute lease holder cancels and joins its watch, reconciliation, and deadline tasks. WatchSandboxStream owns its producer handle and aborts it when the response stream is dropped. Other subsystems retain handles and abort on owner drop. These demonstrate ownership at a narrow scope, but the gateway lacks a common hierarchy and shutdown contract.

Affected Components

Component	Key Files	Role
Gateway lifecycle	crates/openshell-server/src/lib.rs	Constructs workers and listeners, initiates shutdown, and defines cross-subsystem ordering.
Connection serving	crates/openshell-server/src/lib.rs, crates/openshell-server/src/multiplex.rs	Accepts sockets and serves long-lived HTTP/1, HTTP/2, gRPC, upgrade, and WebSocket traffic.
Durable mutation execution	crates/openshell-server/src/grpc/mutation_replay.rs	Detaches opted-in state mutations from request cancellation and can leave an unresolved durable admission if process shutdown interrupts execution.
Supervisor sessions and relays	crates/openshell-server/src/supervisor_session.rs	Publishes persisted ownership, runs long-lived sessions, creates relay pumps, retries endpoint invalidation, and currently carries the local lifetime barrier.
Compute lifecycle	crates/openshell-server/src/compute/mod.rs	Runs watcher/reconciler tasks and must drain admitted lifecycle work before snapshotting and stopping local compute.
Streaming and service routing	crates/openshell-server/src/grpc/sandbox.rs, crates/openshell-server/src/service_routing.rs, crates/openshell-server/src/ws_tunnel.rs	Spawns response producers and bidirectional relay/upgrade pumps whose lifetime is currently tied to channels, guards, or local aborts.
Background maintenance	crates/openshell-server/src/provider_refresh.rs, crates/openshell-server/src/ssh_sessions.rs, crates/openshell-server/src/sandbox_watch.rs, crates/openshell-server/src/readiness.rs, crates/openshell-server/src/tls.rs	Runs periodic work and auxiliary listeners with inconsistent cancellation and joining behavior.
Lifecycle documentation	architecture/gateway.md, docs/reference/sandbox-compute-drivers.mdx, skills/debug-openshell-cluster/SKILL.md	Describes shutdown ordering, workload continuity, and operator diagnostics.

Technical Investigation

Architecture Overview

The gateway currently has one process-level watch<bool> shutdown channel. run_server passes receivers to selected components, signals it after the OS shutdown signal, awaits only the accept-loop task, invokes compute cleanup, and then invokes the supervisor-session drain introduced by #3547. Other tasks either observe the signal without being joined or never observe it.

The required ordering is more specific than a single “cancel everything, then wait” operation:

1. Stop accepting new sockets and reject new state-changing work on existing multiplexed connections.
2. Let already admitted lifecycle mutations reach a stable result before compute cleanup captures its persisted inventory. This is the unresolved race documented by #2826.
3. Stop gateway-managed local compute while supervisor sessions can still report lifecycle state. Kubernetes-owned workloads must continue running.
4. Close supervisor sessions and wait through conditional persisted-owner deletion. This is the guarantee supplied locally by #3547 for #3546.
5. Stop or drain remaining connection pumps and recurring workers, with bounded deadlines and error reporting.

A flat task tracker alone does not encode these dependencies. The model needs scoped ownership or task classes with explicit admission and cancellation phases. It must also avoid circular waits: for example, a connection task may own the gRPC stream that a tracked supervisor-session task needs to observe closing.

Code References

Location	Description
crates/openshell-server/src/lib.rs:183	Extension-token rotation is an unbounded spawned loop with no shutdown input or retained handle.
crates/openshell-server/src/lib.rs:902	The health listener is spawned and its handle is discarded.
crates/openshell-server/src/lib.rs:922	The metrics listener is spawned and its handle is discarded.
crates/openshell-server/src/lib.rs:951	The TLS reload worker receives the shutdown channel and returns a handle, but the caller discards the handle.
crates/openshell-server/src/lib.rs:960	The main listener is the only top-level server task retained and joined by run_server.
crates/openshell-server/src/lib.rs:1003	Store polling, SSH reaping, relay reaping, and provider refresh are started through independent fire-and-forget APIs.
crates/openshell-server/src/lib.rs:1015	Shutdown signals the global channel, joins the accept loop, runs compute cleanup, then drains supervisor sessions.
crates/openshell-server/src/lib.rs:1042	serve_gateway_listener stops the accept loop on shutdown but does not own accepted connection tasks.
crates/openshell-server/src/lib.rs:1142	spawn_gateway_connection detaches each accepted connection and returns no handle; an accepted HTTP/2 connection can host multiple later streams.
crates/openshell-server/src/multiplex.rs:231	Hyper owns all HTTP/gRPC streams for one connection until serve_connection_with_upgrades returns.
crates/openshell-server/src/grpc/mutation_replay.rs:40	A global semaphore bounds detached mutation execution concurrency but provides no shutdown admission gate or drain.
crates/openshell-server/src/grpc/mutation_replay.rs:176	The mutation task is spawned before durable admission and awaited by the handler, protecting caller cancellation but not process shutdown.
crates/openshell-server/src/supervisor_session.rs:320	PR #3547 adds admission state, a shutdown signal, and an RwLock<()> lifetime barrier to the session registry.
crates/openshell-server/src/supervisor_session.rs:363	Session tracking acquires a read guard before checking admission, preventing a shutdown race.
crates/openshell-server/src/supervisor_session.rs:382	Session shutdown signals loops and waits up to ten seconds for the lifetime write guard.
crates/openshell-server/src/supervisor_session.rs:951	The pending-relay reaper is an unbounded detached loop without cancellation.
crates/openshell-server/src/supervisor_session.rs:1058	RelayStream creates detached inbound and outbound pump tasks.
crates/openshell-server/src/supervisor_session.rs:1646	Peer relay bridges create paired detached pumps.
crates/openshell-server/src/supervisor_session.rs:1803	Supervisor setup is spawned and immediately joined so caller cancellation cannot interrupt post-publication cleanup.
crates/openshell-server/src/supervisor_session.rs:1973	The long-lived session task owns cleanup and the local lifetime guard; it also starts endpoint-status retry work.
crates/openshell-server/src/grpc/policy/endpoint_status.rs:459	Endpoint-status invalidation retries indefinitely until persistence succeeds or ownership changes.
crates/openshell-server/src/compute/mod.rs:2667	Top-level compute watcher tasks receive shutdown but their handles are discarded.
crates/openshell-server/src/compute/mod.rs:2701	Compute cleanup snapshots/stops persisted running-intent sandboxes and shuts down a managed driver process.
crates/openshell-server/src/compute/mod.rs:3340	The HA lease holder demonstrates cooperative cancellation followed by joining three owned child tasks.
crates/openshell-server/src/grpc/sandbox.rs:80	WatchSandboxStream demonstrates response-scoped ownership by retaining and aborting its producer task on drop.
crates/openshell-server/src/service_routing.rs:495	WebSocket upgrades and upstream HTTP connection drivers are detached from gateway shutdown.
crates/openshell-server/src/readiness.rs:99	Database health polling is explicitly documented as living until runtime teardown.
architecture/gateway.md:953	Current architecture documentation specifies the #3547 session-specific drain but no general task hierarchy.

Current Behavior

When shutdown begins, the listener watch signal only terminates TcpListener::accept. Existing connection tasks remain live, and multiplexed HTTP/2 clients can still create streams until their connection future ends. Most RPC work runs inside those connection futures, while selected operations spawn child tasks to survive request cancellation or drive response streams.

The compute stop sweep runs after the accept loop has ended but before connection tasks have drained. Consequently, a create or lifecycle mutation admitted earlier can commit after the sweep captures the sandbox inventory, as described by #2826. The durable mutation adapter narrows caller-cancellation races but does not solve process ownership: its task is bounded by a semaphore rather than registered with shutdown.

PR #3547 solves the supervisor-owner manifestation locally. It closes only supervisor-session admission, uses a read guard to track setup and cleanup, signals session loops after compute cleanup, and takes the write guard as a drain barrier. Relay pumps, endpoint-status retry work, and other gateway tasks remain outside that barrier.

Finally, run_server returns. The CLI shuts down tracing and main exits, dropping the Tokio runtime and every remaining future. Dropping futures releases in-memory guards, but it cannot complete awaited persistence or domain cleanup.

What Would Need to Change

The gateway needs a named owner for asynchronous work rather than direct tokio::spawn at subsystem boundaries. That owner must support admission closure and bounded draining without requiring every short-lived helper task to become process-global. Subsystems should register only work whose lifetime crosses a request or owner boundary, while locally scoped pumps can remain owned by their parent connection or response object.

The lifecycle orchestrator needs explicit phases rather than one undifferentiated shutdown signal. Connection/RPC admission, durable mutation execution, compute watchers and cleanup, supervisor sessions, relay work, and maintenance loops have different cancellation and completion requirements. Each spawn site must be classified accordingly, and spawn APIs should return or register ownership rather than silently detach.

The supervisor-session registry should migrate from the RwLock<()> barrier to the shared abstraction only after equivalent “register before durable publication, reject after closure, clean through owner deletion” behavior is demonstrated. The lifecycle-mutation path must similarly establish admission before persistence and drain before compute inventory. Existing conditional updates and replacement-owner protections remain domain logic and should not move into a generic task tracker.

Alternative Approaches Considered

1. Keep adding subsystem-local lifetime barriers. This minimizes each patch, as #3547 does, but duplicates admission, timeout, error, and lock-order reasoning. It does not solve cross-subsystem ordering such as mutation drain before compute cleanup.
2. Track only accepted network connections. Joining connection futures would cover ordinary RPCs, but not enough on its own: existing HTTP/2 connections need an admission boundary, and tasks intentionally detached from request cancellation can outlive the connection future.
3. Install one flat process-wide task tracker. This centralizes joining but cannot by itself express which tasks should finish, which should be cancelled, or the required ordering between mutation, compute, and supervisor cleanup. Forever loops would deadlock a drain unless cancellation is coordinated first.
4. Use subsystem-owned task scopes coordinated by the gateway lifecycle. This best matches current boundaries and allows phased close/cancel/drain behavior. Human review is needed to decide whether this uses a shared internal abstraction, Tokio JoinSet ownership through manager tasks, a task-tracker dependency, or a small repository-local implementation.
5. Rely on repeated cleanup sweeps or longer shutdown sleeps. This cannot close admission races and only changes their probability; #2826 already explains why a final inventory can always be overtaken by another mutation.

Patterns to Follow

• Preserve the compute lease holder's pattern of signaling cancellation and awaiting child handles (compute/mod.rs:3340-3401).
• Preserve response-scoped ownership where the response object is the natural parent, as in WatchSandboxStream (grpc/sandbox.rs:80-124).
• Follow the structural-ownership principle in #3154: an abstraction should make registration, completion, and cleanup inseparable and should support deterministic race tests.
• Preserve the #3547 invariant that tracking is established before admission can publish durable ownership.
• Preserve compare-and-swap and identity checks in supervisor-owner deletion; task ownership must not weaken replacement-session safety.
• Keep shutdown bounded and emit actionable diagnostics identifying the phase and outstanding task class.
• Keep local-compute shutdown separate from Kubernetes control-session shutdown so Kubernetes workloads continue running.

Proposed Approach

Introduce subsystem-owned asynchronous task scopes coordinated by a small gateway lifecycle orchestrator. Each scope should expose close-admission, cooperative-cancel, and bounded-drain operations, while tasks remain registered from before their first durable side effect through their final required cleanup. Define shutdown as ordered phases so admitted mutations drain before compute inventory, compute cleanup runs while supervisor reporting is available, and session ownership drains afterward. Classify connection and response child tasks by their actual parent rather than forcing every helper into a global registry. Once the common model proves the #3546 guarantees, remove the supervisor-specific lifetime lock.

Scope Assessment

• Complexity: High
• Confidence: Medium — the failure modes and ordering constraints are established, but ownership granularity and the shared abstraction require design review.
• Estimated files to change: 8-15 across gateway lifecycle, connection serving, mutation execution, supervisor sessions, compute coordination, tests, and documentation.
• Issue type: refactor
• Gateway config impact: No configuration change is required for the initial design. If phase deadlines become operator-configurable, update docs/reference/gateway-config.mdx, Helm rendering, and deployment examples.
• LSM impact: None expected. The change concerns Tokio task ownership and shutdown coordination; it does not alter process identity, /proc access, file labels, binary execution, or inter-process visibility under SELinux or AppArmor.

Risks & Open Questions

• Should all long-lived gateway tasks use one shared task-scope implementation, or should the lifecycle orchestrator coordinate independent subsystem-specific managers behind a common interface?
• Which task classes must finish gracefully, which may be cancelled once their caller is gone, and which may be aborted only after a deadline?
• Should shutdown use one global deadline budget or separate budgets per phase? If configurable, what operator-facing contract and defaults should apply?
• How should unresolved durable mutation admissions be recovered if the process is force-killed after admission but before completion, beyond what graceful draining can guarantee?
• How should Hyper HTTP/2 graceful shutdown interact with an application-level mutation admission gate so read-only requests can finish without allowing new lifecycle work?
• How should shutdown errors from compute cleanup, session cleanup, background workers, and task panics be aggregated without hiding the earliest safety-relevant failure?
• Can the migration be incremental without running two competing admission authorities for the same task class?
• The change affects cross-platform server code. Windows compile and native gateway checks must remain green even though Docker, Podman, Kubernetes, and VM runtime tests remain platform-specific.

Disposition Readiness

• State: state:validated
• Assessment: Two independently documented shutdown races (#2826 and #3546) establish the problem, and the current source inventory identifies the ownership gaps and ordering constraints needed for a human accept/decline decision. PR #3547 supplies a tested local mitigation and a clear migration target.
• Missing evidence: None for disposition. Implementation still requires a human decision on task-scope granularity, cancellation policy, and deadline semantics.

Test Considerations

• Add a deterministic gateway-level test for #2826: pause an admitted sandbox create before persistence, start shutdown, release the mutation, and prove compute cleanup includes the resulting running-intent sandbox.
• Retain or migrate the five #3547 supervisor-session tests covering admission races, post-registry cleanup, replacement ownership, timeout, and an empty drain.
• Add connection-level tests showing shutdown stops new mutation admission on an already accepted HTTP/2 connection while allowing the chosen class of in-flight requests to complete.
• Add task-scope unit tests for registration racing with close, late registration rejection, task panic reporting, cooperative cancellation, deadline expiry, and no missed completion notifications.
• Add background-worker tests proving shutdown joins workers and prevents post-drain store writes or retries.
• Add multi-replica coverage for reconciler lease release and supervisor owner replacement where the shutdown phases depend on shared persistence.
• Run the existing Docker, Podman, VM, and Kubernetes E2E shutdown/restart paths. VM must retain the controlled delayed-owner-release regression from #3546; Kubernetes must show workloads continue while control sessions reconnect.
• Update listener tests, which currently await only the accept-loop handle, to assert accepted connection-task behavior during shutdown.

---

Created by spike investigation. state:validated means the issue is ready for human disposition. A human applies state:accepted or places the issue on the roadmap if OpenShell should pursue the work. To queue unattended agent planning, a human applies agent:plan-requested; on a direct request, an agent may use build-from-issue after warning about missing expected workflow labels. This issue incorporates the still-valid race from closed stale issue #2826 and the local mitigation in #3547; it does not reopen acceptance or roadmap decisions automatically.