Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

bug: Kubernetes bootstrap retry limits can leave sandboxes terminal during gateway startup

Offen
  • Stimmt etwas nicht?
#3,274 0 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
45/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
kubernetes, rust
Bereich
infrastructure

Rechercherichtung

Beginne mit crates/openshell-sandbox/src/lib.rs, insbesondere mit der grpc_retry-Logik zum Abrufen der Richtlinie und SIDECAR_READY_TIMEOUT_SECS, und untersuche anschließend crates/openshell-driver-kubernetes/src/driver.rs sowie crates/openshell-server/src/compute/mod.rs. Erstelle die vorgeschlagene isolierte Kubernetes-Reproduktion mit verzögerter Gateway-Verfügbarkeit und verifiziere die Lebenszyklusübergänge, die Beibehaltung des Workspace, begrenzte abbrechbare Retries und einen dokumentierten Wiederherstellungspfad, ohne abgeschlossene Workloads erneut auszuführen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

state:triage-needed

User Story

As a Kubernetes operator, I want sandbox bootstrap to tolerate temporary gateway unavailability, so that dependency startup ordering does not require destructive recovery.

Problem Statement

This is a source-based report against public OpenShell v0.0.116 code. The deterministic reproduction below is proposed, not a claim of a completed isolated upstream test. No private deployment logs or configuration are included.

Three public implementation details appear to make delayed gateway startup a terminal sandbox failure:

  1. grpc_retry in the sandbox runtime makes five attempts, with 1/2/4/4-second waits. When initial policy fetching cannot connect, it returns an error after exhausting those attempts.
  2. The same file defines a separate SIDECAR_READY_TIMEOUT_SECS = 120 for the process supervisor waiting on the network supervisor. Extending the policy retry window alone would not remove this second deadline.
  3. The Kubernetes driver deliberately renders restartPolicy: Never, with a comment explaining that kubelet must not replace the canonical main-process generation after supervisor exit.

Consequently, a transient dependency outage before workload activation can exhaust bootstrap retries, terminate the pod, and leave no automatic retry when the gateway becomes available. The same retry limits remain in inspected main commit 02b664bb0d978ac0baec9aaa0bf06a2a4f67e83d.

Impact / Why This Matters

Kubernetes does not guarantee that the gateway is ready before sandbox pods start during cluster recovery. A short-lived startup outage should not require deleting a persistent sandbox.

The v0.0.116 gateway start handler rejects terminal Error phases; it allows Ready as an idempotent return and Stopped/Starting for start. This makes a documented, data-preserving recovery path important in addition to bootstrap retry behavior. Blanket automatic deletion/recreation would be inappropriate for persistent workspaces.

Acceptance Criteria

  • Before workload activation, a sandbox can tolerate transient gateway unavailability and become usable once the gateway returns without manual deletion.
  • A regression test delays gateway availability beyond both the current policy-fetch retry window and the 120-second process-sidecar wait.
  • No workload starts before authenticated policy and required enforcement bootstrap succeed. Permanent authentication/configuration/policy errors remain explicit failures, and TLS validation is preserved.
  • Individual attempts are bounded, retries use capped backoff, and waiting is cancellable and diagnosable.
  • Existing workspace data survives the tested recovery.
  • Explicit stops and completed workloads remain terminal; fixing bootstrap does not silently replay completed commands.
  • Document a supported, data-preserving recovery path for sandboxes already in terminal bootstrap error.

Reproduction Steps

Proposed deterministic test, not yet executed as an isolated upstream reproduction:

  1. Use a disposable Kubernetes deployment with the v0.0.116 gateway, supervisor, and sidecar topology.
  2. Create a persistent sandbox and write a workspace sentinel.
  3. Simulate dependency recovery so a replacement sandbox pod starts while its gateway callback is unavailable. Keep the gateway unavailable for more than 120 seconds.
  4. Restore gateway availability.
  5. Check pod/Sandbox/gateway state, whether a shell becomes usable without resource deletion, and whether the workspace sentinel is retained.

The public code predicts an exhausted Policy fetch retry followed by the separate sidecar-control-socket timeout. A regression test should establish the exact lifecycle transitions as well as successful recovery after the fix. Fault injection should use an isolated test deployment, not a shared production gateway.

Environment

Source versions and configuration relevant to the proposed test, not private deployment inventory:

  • OpenShell: public tag v0.0.116, latest published release at filing.
  • Additional source inspection: main commit 02b664bb0d978ac0baec9aaa0bf06a2a4f67e83d; no runtime test of that commit is claimed.
  • Compute driver: Kubernetes.
  • Supervisor topology: sidecar.
  • Persistent workspace; driver-generated restartPolicy: Never.

Related Work

#2179 and #2269 concern Docker/Podman Error-phase recovery. #2417 explicitly keeps Kubernetes lifecycle cluster-owned. They provide useful lifecycle context but do not appear to resolve the Kubernetes bootstrap retry limits above.

Changing pod restartPolicy globally to Always would alter the deliberately terminal main-process contract. This report requests bootstrap-specific resilience and a supported recovery path, not automatic replay or deletion of arbitrary failed workloads.

Vorherrschende Sprache
Rust
Sterne
15.4k
Forks
1.7k
Ø Merge
1 T. 21 Std.
Gemergte PRs (30 T.)
366

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus NVIDIA/OpenShell

Alle Issues in NVIDIA/OpenShell

Ähnliche Issues

Weitere Issues zu Rust

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.