bug(supervisor-network): boundary reconnect is followed by proxy exit and ControlSupervisorExited

Bắt đầu bằng cách lần theo chuỗi khôi phục boundary_server được ghi log, vòng lặp accept của proxy trung gian mạng, supervisor_session và đường dẫn chờ của trình điều khiển Docker. Tái hiện việc mất boundary hoặc tiêm lỗi khi hoạt động polling của môi trường provider đang diễn ra, sau đó thêm phạm vi kiểm thử hồi quy được mô tả trong các tiêu chí nghiệm thu. Được xem là hoàn tất khi các phiên bản được hỗ trợ vẫn hoạt động bình thường, các phiên bản không tương thích thất bại trước khi khởi chạy và các lỗi vẫn giữ lại thông tin lỗi boundary hoặc supervisor có thể dùng để xử lý.

User Story

As an operator running a long-lived agent sandbox through OpenShell's RFC 0012 split supervisor/workload architecture, I need a transient Sandbox Protocol boundary failure to recover cleanly—or be rejected as an incompatible deployment before launch—so that an otherwise healthy workload is not terminated and left in Error.

Problem Statement

A Docker-backed sandbox ran successfully for approximately 15 minutes, including allowed proxied HTTPS traffic and successful in-place gateway/Sandbox Protocol credential renewal. The network-mediation proxy then exited with boundary unavailable.

The workload boundary detected the supervisor connection loss, froze the workload, and reported recovery about 4 ms later. Ten seconds after that apparent recovery, the boundary gRPC exchange failed. The supervisor stream ended, the gateway reported an empty Docker wait error, and the sandbox transitioned to Error with reason ControlSupervisorExited. The Docker driver then stopped the workload; the workload container itself was not OOM-killed and exited 0.

Repeated ReportEndpointStatus calls were already returning gRPC status 9 before the fatal event, while GetSandboxConfig and GetSandboxProviderEnvironment continued succeeding. The policy revision remained unchanged, but provider-environment refreshes produced new revision values on successive polls.

The tested compatibility triplet was mixed and is important context:

• CLI/gateway: 0.0.117-dev.160+g9c41f057c
• Supervisor image: metadata-only derivative of the dev.158 supervisor image; the only intended change was adding a private CA trust anchor
• Sandbox runtime image: dev.158

If that combination is unsupported, sandbox creation should fail with an actionable compatibility error. It should not launch successfully and later fail through the boundary protocol.

I searched current issues and found related—but not exact—reports:

• #3308 covers the resulting unrecoverable Error state after a supervisor session ends.
• #3311 covers a network-policy peer-binary resolution failure killing the supervisor; no peer-binary failure or denied request preceded this event.
• #3314 covers a long-lived relay/proxy lifecycle failure, but its sandbox remains incorrectly Ready rather than entering Error.
• #2337 and #2372 cover proxy/SSH accept loops exiting on transient descriptor errors; no EMFILE/ENFILE error was observed here.

Impact / Why This Matters

The sandbox and its agent become unavailable even though the workload was healthy and the proxied request path had been working. The gateway terminates the workload after the control supervisor exits, so active sessions are lost and end-to-end automation cannot remain available.

The current diagnostics do not identify why the boundary became unavailable. The gateway records an empty Docker container wait error, while the earlier repeated endpoint-status failures are logged only as transient. This makes it difficult to distinguish a version-contract violation, boundary protocol failure, or recoverable transport interruption.

No automatic recovery occurred. Recreating the sandbox loses process/session state and prevents this configuration from being qualified for long-lived use.

Acceptance Criteria

• A supported gateway/supervisor/runtime triplet remains healthy under normal allowed proxied traffic and periodic provider-environment refresh.
• A transient Sandbox Protocol connection loss that successfully reconnects does not subsequently terminate the proxy accept loop or workload.
• An unsupported gateway/supervisor/runtime protocol combination is rejected before workload launch with the incompatible components and revisions identified.
• Repeated ReportEndpointStatus failures expose an actionable reason and do not remain an indefinitely ignored precursor to supervisor failure.
• If the control supervisor must terminate, the sandbox condition retains the concrete boundary/supervisor error instead of an empty Docker container wait error.
• A regression test covers boundary loss/recovery while the network-mediation proxy and provider-environment polling are active.

Reproduction Steps

The failure has been observed on a clean host but has not yet been reduced to a deterministic fault-injection trigger.

1. Install OpenShell CLI/gateway 0.0.117-dev.160+g9c41f057c with the Docker driver.
2. Select the dev.158 supervisor and sandbox-runtime images. Use a supervisor derivative that changes only the CA trust store.
3. Create a sandbox using the split supervisor/workload boundary, automatic TLS inspection, Landlock best_effort, and two provider-projected environment values. No OpenShell content middleware is enabled.
4. Start a long-running workload and make a normal policy-allowed proxied HTTPS request through host.openshell.internal.
5. Leave the sandbox running. In the observed run, endpoint-status reports repeatedly returned gRPC status 9, followed after approximately 15 minutes by the boundary/proxy failure shown below.
6. Inspect the sandbox: it transitions to Error with Ready=False, reason ControlSupervisorExited.

Environment

• OpenShell CLI/gateway: 0.0.117-dev.160+g9c41f057c
• Supervisor: dev.158 image, locally derived only to add a CA trust anchor
• Sandbox runtime: dev.158 image
• Compute driver: Docker
• Docker: 29.8.0
• OS: Ubuntu 24.04, x86_64
• Kernel: 7.0.0-1012-aws
• Policy/runtime: RFC 0012 split supervisor/workload, automatic TLS inspection, Landlock best_effort
• Integration: APF-projected provider profiles with AgentGateway/ATGW; no OpenShell content middleware
• Workload container after failure: OOMKilled=false, exit code 0 after the Docker driver stopped it
• Latest tested build: dev.160. This RFC 0012 development layout was not destructively replaced with the older stable release solely for comparison.

Logs

# Repeated before the fatal event; config/provider fetches continued to succeed.
20:34:20.122Z ReportEndpointStatus http.response.status_code=200 rpc.grpc.status_code=9 otel.status_code="ERROR"
20:34:27.115Z GetSandboxConfig http.response.status_code=200
20:34:27.121Z GetSandboxProviderEnvironment completed successfully provider_count=2 env_count=2
20:34:27.124Z ReportEndpointStatus http.response.status_code=200 rpc.grpc.status_code=9 otel.status_code="ERROR"

# Fatal sequence.
20:34:35.799Z OCSF NET:FAIL [HIGH] [msg:Network-mediation source failed; proxy accept loop exiting: boundary unavailable: read boundary control response header: ...]
20:34:35.801Z WARN boundary_server::linux: Sandbox Protocol connection lost; workload frozen pending authenticated recovery
20:34:35.801Z OCSF FINDING:CREATE [MED] "Sandbox Supervisor Connection Lost" [type:sandbox-supervisor-connection-lost confidence:high]
20:34:35.805Z INFO boundary_server::linux: Sandbox Protocol connection recovered; workload resumed
20:34:45.858Z WARN boundary_server::linux: Boundary gRPC exchange failed
20:34:45.885Z INFO supervisor_session: supervisor session: stream closed by supervisor
20:34:45.885Z INFO supervisor_session: supervisor session: ended
20:34:45.886Z PushSandboxLogs http.response.status_code=200 rpc.grpc.status_code=13 latency_ms=909920
20:34:47.016Z WARN openshell_driver_docker: Failed to wait for Docker supervisor container error="Docker container wait error:"
20:34:47.032Z INFO compute: Sandbox phase changed old_phase=Provisioning new_phase=Error
20:34:47.032Z WARN compute: Sandbox failed to become ready reason=ControlSupervisorExited
20:34:47.046Z ERROR network_broker: sandbox network broker listener failed
20:34:47.109Z INFO openshell_driver_docker: Stopped Docker sandbox after control supervisor failure

# Workload container state after the driver stopped it.
state=exited exit=0 oom=false

All hostnames, sandbox/container identifiers, image digests, private endpoints, credentials, and enterprise payloads have been removed. Full sanitized logs can be provided if needed.