VM driver (v0.0.116, Linux x86_64, nested KVM): sandbox never reaches Ready - PushSandboxLogs stream dropped with UnexpectedEof during boot

Start by tracing the VM guest supervisor's PushSandboxLogs stream and the gateway request handling, using the provided nested-KVM configuration and debug logs; inspect the openshell-driver-vm --internal-run-vm path. Done means sandbox create reaches Ready, PushSandboxLogs succeeds or cannot block provisioning, and the Landlock limitation is addressed or documented.

User Story

As an operator running the OpenShell gateway with the experimental vm compute driver on a Linux VM with nested KVM, I want openshell sandbox create to bring a sandbox to Ready so that I can run per-task workloads in microVM isolation instead of falling back to the podman driver.

Problem Statement

On a fresh 0.0.116 deb install, every sandbox create with the vm driver hangs in Provisioning and times out after 300 s ("Last reported status: Starting: VM is starting"). Gateway debug logs show the guest completes the mTLS handshake (TLS 1.3, ALPN h2, client-cert auth) and several unary RPCs (ReportPolicyStatus, GetSandboxConfig, UpdateConfig, GetSandboxProviderEnvironment, GetInferenceBundle) cleanly, but the one client-streaming call, PushSandboxLogs, has its connection dropped mid-stream with a raw TCP EOF (UnexpectedEof, no TLS close_notify) and returns gRPC status 13. After that the guest supervisor makes no further progress; it does not crash, does not retry, and the openshell-driver-vm --internal-run-vm process stays alive indefinitely. The failure is identical for a sub-second main process and for sleep infinity left running 14+ minutes, so it is not the short-lived-main-process case in #3117. Ruled out: gateway hostname (host.openshell.internal is set and the handshake succeeds), driver_dir (corrected to /usr/libexec/openshell, driver connects with in_tree=true), TLS/cert config, permissions/sudo, state_dir. Secondary: the guest logs two [HIGH] "Landlock Filesystem Sandbox Unavailable" findings on every boot, so landlock.compatibility: hard_requirement could not be satisfied under this driver/image even if boot completed.

Impact / Why This Matters

The vm driver is unusable on this platform class (Linux guest under a hypervisor with nested KVM), which is a common lab and homelab shape. Operators who need microVM isolation for untrusted agent workloads must fall back to container isolation (podman driver), which also means losing the guest-kernel boundary. Because the guest reports Landlock unavailable, the fallback is currently the only way to get hard_requirement enforcement at all on this host.

Acceptance Criteria

• With the configuration below, openshell sandbox create --name hello --detach -- sleep infinity reaches Ready within normal image-pull plus boot time, and openshell sandbox exec -n hello -- uname -r returns the guest kernel version.
• PushSandboxLogs completes (gRPC status 0) during boot, or a failure of that stream is retried/non-fatal so provisioning can continue.
• Either Landlock is available inside the community VM-driver guest kernel, or the documentation states that hard_requirement is not satisfiable under the VM driver.

Reproduction Steps

1. Install openshell_0.0.116-1_amd64.deb (checksum-verified) on the host described below; run the gateway as a systemd user service with the config in Environment.
2. openshell gateway add https://127.0.0.1:17670 --local
3. openshell sandbox create --no-keep -- sh -c "echo hello-from-microvm; uname -r" → times out after 300 s in Provisioning.
4. To rule out a short-lived main process: openshell sandbox create --name hello --detach -- sleep infinity, then poll openshell sandbox get hello -o json every 15 s for 14+ minutes → phase stays Provisioning; openshell sandbox exec -n hello -- sh -c "uname -r" → "sandbox 'hello' is not ready (phase: Provisioning)".
5. ps shows openshell-driver-vm --internal-run-vm … alive throughout; the sandbox's sleep infinity never exited.
6. Switch the same gateway to compute_drivers = ["podman"] (rootless socket, grpc_endpoint = "https://host.containers.internal:17670"), no other change → sandboxes reach Ready immediately and exec works.

Environment

• OpenShell 0.0.116 (official GitHub release deb); driver binaries at /usr/libexec/openshell.
• Host: Ubuntu 26.04.1, kernel 7.0.0-30-generic, x86_64, a guest VM under Proxmox VE with nested KVM enabled (kvm_amd loaded, /dev/kvm usable by the operator, member of group kvm); 8 vCPU, 22 GB RAM.
• Rootless Podman 5.7.0 (used for image pulls; the podman driver works on the same host).
• openshell-gateway as a systemd user service, mTLS, bind_address = "127.0.0.1:17670".
• Gateway config:

[openshell.gateway]
bind_address    = "127.0.0.1:17670"
compute_drivers = ["vm"]

[openshell.drivers.vm]
driver_dir       = "/usr/libexec/openshell"
default_image    = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
bootstrap_image  = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
grpc_endpoint    = "https://host.openshell.internal:17670"
vcpus            = 2
mem_mib          = 3072
overlay_disk_mib = 4096
krun_log_level   = 1

Logs

Guest console, last lines before it goes silent:


2026-09-16T05:08:41.730Z OCSF CONFIG:LOADED [INFO] Acknowledged initial policy revision as loaded [version:1] ...
2026-09-16T05:08:41.745Z OCSF FINDING:CREATE [HIGH] "Landlock Filesystem Sandbox Unavailable" (x2)
2026-09-16T05:08:41.789Z OCSF CONFIG:CLEANED_UP [INFO] Network namespace cleaned up [ns:sandbox-2194cf63]


Gateway (debug) for the failing call; the handshake for this same connection had already completed with client auth:


2026-09-16T05:11:54.194Z DEBUG log: Attempting client auth
2026-09-16T05:11:54.951095Z DEBUG Connection{peer=Server}: h2::proto::connection: Connection::poll; IO error error=UnexpectedEof
2026-09-16T05:11:54.951209Z DEBUG request{...path="/openshell.v1.OpenShell/PushSandboxLogs"...}: tonic::codec::decode: decoder inner stream error: Status { code: Unknown, message: "h2 protocol error: error reading a body from connection", source: Some(hyper::Error(Body, Error { kind: Io(Custom { kind: UnexpectedEof, error: "peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof" }) })) }
2026-09-16T05:11:54.951359Z  INFO request{...rpc.method="PushSandboxLogs" http.response.status_code=200 rpc.grpc.status_code=13 otel.status_code="ERROR"}: response status=200 latency_ms=615


CLI:


Error:   × sandbox provisioning timed out after 300s. Last reported status: Starting: VM is starting


Happy to provide the full debug-level gateway journal, the complete guest `rootfs-console.log`, and a run with `krun_log_level` raised.