Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

feat(ha): add production scaling signals and graceful gateway redistribution

Ouverte
  • Quelque chose ne va pas ?
#3,528 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@EmilienM y travaille déjà.

Depuis le 2/10/2026.

  • #4147 par @EmilienM — ouverte

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
25/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
helm, kubernetes, postgresql, rust
Domaine
backend-api-design, databases, devops, distributed-systems, observability

Piste de recherche

Start with PR #1868 and its linked discussions on graceful redistribution, scaling metrics, advisory-lock serialization, and WatchSandbox polling. Trace the gateway lifecycle, supported metrics endpoint, Helm documentation, mutation locking, and cross-replica watch synchronization. Done means the acceptance criteria and load or integration tests cover all four production-scale behaviors.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

User Story

As an OpenShell cluster operator, I want multi-replica gateways to expose actionable capacity signals and redistribute supervisor ownership during planned changes, so that scaling and rolling upgrades do not leave one replica overloaded while new replicas remain idle.

Problem Statement

PR #1868 makes multi-replica gateway routing possible, but several production-scale behaviors remain incomplete:

  • Supervisor ownership stays with the original connection. Rolling restarts and failover can leave the last surviving replica holding most sessions until sandboxes naturally reconnect.
  • The gateway does not expose held-session, pending-relay, or peer-relay rate and latency metrics suitable for capacity planning or autoscaling.
  • Cross-object mutations use one fleet-wide PostgreSQL advisory-lock key, so unrelated sandbox and provider mutations serialize across all replicas.
  • WatchSandbox performs one database lookup per actively watched sandbox, per polling interval, per replica.

Impact / Why This Matters

Operators can add gateway replicas, but cannot tell when capacity is exhausted or rely on replicas receiving a balanced share of established supervisor sessions. The current workaround is to overprovision, manually inspect logs, and wait for sandbox churn or force reconnects after a rollout. That does not prevent pending-relay exhaustion, database amplification, or fleet-wide mutation contention at larger scale.

Proposed Design

Provide an observable, documented HA lifecycle with graceful draining and bounded shared-database load:

  • A terminating gateway stops accepting new ownership, releases or transfers existing ownership, and prompts affected supervisors to reconnect before the termination grace period expires.
  • Gateway metrics expose held supervisor sessions, pending relays versus capacity, and peer-relay request rate, failure rate, and latency.
  • The Helm chart documents those signals and provides an optional autoscaling configuration, or a documented integration contract for a Kubernetes metrics adapter.
  • Unrelated object mutations do not share one fleet-wide serialization point when their invariants cannot overlap.
  • Cross-replica sandbox watches use a batched or notification-based mechanism whose database work does not grow as one query per watched sandbox on every replica.

Acceptance Criteria

  • During a graceful gateway rollout, established supervisor ownership redistributes without waiting for natural sandbox churn, within the configured termination grace period.
  • Operators can observe per-replica held sessions, pending relays/capacity, and peer-relay rate, failures, and latency through the supported metrics endpoint.
  • The Helm documentation explains how to use these signals for horizontal scaling; any chart-provided HPA behavior is optional and configurable.
  • Mutations for unrelated sandboxes or providers can proceed concurrently across replicas unless they share a documented cross-object invariant.
  • Cross-replica watch synchronization avoids one database query per watched sandbox per interval per replica.
  • Load or integration tests cover rollout redistribution, relay saturation signals, concurrent unrelated mutations, and large watch sets.
  • HA architecture and Kubernetes operator documentation describe the lifecycle, capacity signals, and database-scaling behavior.

Alternatives Considered

Keeping ownership sticky is simple and preserves correctness, but leaves rollout skew entirely dependent on sandbox churn. Relying only on logs does not provide stable autoscaling inputs. Increasing the PostgreSQL connection pool masks neither fleet-wide lock serialization nor query amplification.

Agent Investigation

This groups the remaining production-scale review discussions from PR #1868:

Related umbrella issue: #1021.

Langage dominant
Rust
Étoiles
15.4k
Forks
1.7k
Merge moyen
1 j 19 h
PR mergées (30 j)
347

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de NVIDIA/OpenShell

Toutes les issues de NVIDIA/OpenShell

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.