Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

bug: Kubernetes bootstrap retry limits can leave sandboxes terminal during gateway startup

Ouverte
  • Quelque chose ne va pas ?
#3,274 0 commentaires 1 réaction 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
45/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
kubernetes, rust
Domaine
infrastructure

Piste de recherche

Commencez par crates/openshell-sandbox/src/lib.rs, en particulier la logique grpc_retry de récupération de la policy et SIDECAR_READY_TIMEOUT_SECS, puis examinez crates/openshell-driver-kubernetes/src/driver.rs et crates/openshell-server/src/compute/mod.rs. Mettez en place la reproduction Kubernetes isolée proposée avec une disponibilité retardée de la gateway et vérifiez les transitions du cycle de vie, la conservation du workspace, les retries bornés et annulables, ainsi qu’un chemin de récupération documenté sans rejouer les workloads terminés.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

state:triage-needed

User Story

As a Kubernetes operator, I want sandbox bootstrap to tolerate temporary gateway unavailability, so that dependency startup ordering does not require destructive recovery.

Problem Statement

This is a source-based report against public OpenShell v0.0.116 code. The deterministic reproduction below is proposed, not a claim of a completed isolated upstream test. No private deployment logs or configuration are included.

Three public implementation details appear to make delayed gateway startup a terminal sandbox failure:

  1. grpc_retry in the sandbox runtime makes five attempts, with 1/2/4/4-second waits. When initial policy fetching cannot connect, it returns an error after exhausting those attempts.
  2. The same file defines a separate SIDECAR_READY_TIMEOUT_SECS = 120 for the process supervisor waiting on the network supervisor. Extending the policy retry window alone would not remove this second deadline.
  3. The Kubernetes driver deliberately renders restartPolicy: Never, with a comment explaining that kubelet must not replace the canonical main-process generation after supervisor exit.

Consequently, a transient dependency outage before workload activation can exhaust bootstrap retries, terminate the pod, and leave no automatic retry when the gateway becomes available. The same retry limits remain in inspected main commit 02b664bb0d978ac0baec9aaa0bf06a2a4f67e83d.

Impact / Why This Matters

Kubernetes does not guarantee that the gateway is ready before sandbox pods start during cluster recovery. A short-lived startup outage should not require deleting a persistent sandbox.

The v0.0.116 gateway start handler rejects terminal Error phases; it allows Ready as an idempotent return and Stopped/Starting for start. This makes a documented, data-preserving recovery path important in addition to bootstrap retry behavior. Blanket automatic deletion/recreation would be inappropriate for persistent workspaces.

Acceptance Criteria

  • Before workload activation, a sandbox can tolerate transient gateway unavailability and become usable once the gateway returns without manual deletion.
  • A regression test delays gateway availability beyond both the current policy-fetch retry window and the 120-second process-sidecar wait.
  • No workload starts before authenticated policy and required enforcement bootstrap succeed. Permanent authentication/configuration/policy errors remain explicit failures, and TLS validation is preserved.
  • Individual attempts are bounded, retries use capped backoff, and waiting is cancellable and diagnosable.
  • Existing workspace data survives the tested recovery.
  • Explicit stops and completed workloads remain terminal; fixing bootstrap does not silently replay completed commands.
  • Document a supported, data-preserving recovery path for sandboxes already in terminal bootstrap error.

Reproduction Steps

Proposed deterministic test, not yet executed as an isolated upstream reproduction:

  1. Use a disposable Kubernetes deployment with the v0.0.116 gateway, supervisor, and sidecar topology.
  2. Create a persistent sandbox and write a workspace sentinel.
  3. Simulate dependency recovery so a replacement sandbox pod starts while its gateway callback is unavailable. Keep the gateway unavailable for more than 120 seconds.
  4. Restore gateway availability.
  5. Check pod/Sandbox/gateway state, whether a shell becomes usable without resource deletion, and whether the workspace sentinel is retained.

The public code predicts an exhausted Policy fetch retry followed by the separate sidecar-control-socket timeout. A regression test should establish the exact lifecycle transitions as well as successful recovery after the fix. Fault injection should use an isolated test deployment, not a shared production gateway.

Environment

Source versions and configuration relevant to the proposed test, not private deployment inventory:

  • OpenShell: public tag v0.0.116, latest published release at filing.
  • Additional source inspection: main commit 02b664bb0d978ac0baec9aaa0bf06a2a4f67e83d; no runtime test of that commit is claimed.
  • Compute driver: Kubernetes.
  • Supervisor topology: sidecar.
  • Persistent workspace; driver-generated restartPolicy: Never.

Related Work

#2179 and #2269 concern Docker/Podman Error-phase recovery. #2417 explicitly keeps Kubernetes lifecycle cluster-owned. They provide useful lifecycle context but do not appear to resolve the Kubernetes bootstrap retry limits above.

Changing pod restartPolicy globally to Always would alter the deliberately terminal main-process contract. This report requests bootstrap-specific resilience and a supported recovery path, not automatic replay or deletion of arbitrary failed workloads.

Langage dominant
Rust
Étoiles
13.2k
Forks
1.6k
Merge moyen
1 j 23 h
PR mergées (30 j)
355

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/OpenShell

Toutes les issues de NVIDIA/OpenShell

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.