Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Complete Slurm proxy health integration, draining, telemetry, and overload validation

Aperta
  • Qualcosa non va?
#1,103 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@nabinchha ci sta già lavorando.

Dal 28/10/2026.

  • #1104 di @nabinchha — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

task

Task Summary

Complete the remaining lifecycle integration, observability, and live overload validation from #966. The core streaming, pooled inference proxy was delivered in #967; this issue carries forward the outstanding requirements so #966 can close without losing them.

Related: #1005 (disconnect cancellation and timeout fixes), #1044 (startup readiness and lifecycle diagnostics), #870 (sealed release acceptance).

Current behavior

  • Streaming, connection reuse, deterministic least-connections routing, bounded failover, aggregate backend health, and 429/Retry-After handling are implemented.
  • Startup readiness probes and required-process exit detection exist. During generation, HTTP backend health is not periodically connected to persisted managed readiness when a backend stays alive but becomes unhealthy.
  • The proxy configures a 10-second shutdown timeout. Runtime cleanup sends TERM to all managed steps, waits 0.2 seconds, then escalates to KILL; serving steps are not kept available until proxy draining completes.
  • Existing metrics include requests, attempts, per-backend active/request counts, connection opens/reuse/failures, retry reasons, and final status counts. Request logs include attempt count, upstream outcome sequence, final status, failover usage, and overall elapsed time.
  • Automated tests cover queue admission, 429 failover/header handling, streaming, connection reuse, and failure paths. Live vLLM overload and the resulting client AIMD behavior remain unverified; this is a validation gap, not a confirmed overload defect.

Remaining Work

Runtime health integration
  • Periodically check backend health after startup and propagate required-backend service loss to managed readiness and attempt state.
  • Cover a backend becoming unhealthy while its process remains alive, as well as process exit.
Coordinated shutdown
  • Stop admitting new proxy requests and allow active responses to drain for a bounded window while required backends remain available.
  • Coordinate runtime TERM/KILL escalation with that window, then close remaining sockets and stop serving steps.
  • Cover normal completion, cancellation, and failure.
Observability completion
  • Record the selected upstream for each attempt and separate connection and response timings.
  • Add connection eviction accounting and ensure terminal timeouts are observable, including when no further retry occurs.
  • Retain existing counters; final 429 and 5xx counts are already available through final-status counters.
  • Keep secrets, authorization headers, prompts, and generated content out of logs and metrics.
Live overload validation
  • Exercise actual queue saturation with partial and total replica overload.
  • Demonstrate final 429/Retry-After propagation after bounded failover and the client's admission reduction and cooldown.
  • Record sanitized code/artifact identities, expected and actual output counts, failure/retry counts, elapsed time, records per minute, and proxy CPU/memory telemetry. Coordinate release-artifact evidence with #870.

Acceptance Criteria

  • Required-backend health loss after startup is reflected in persisted managed readiness and the defined attempt-state behavior, even while its process remains alive.
  • Cleanup permits active proxy responses to complete within the configured drain budget before stopping their backends, then terminates remaining requests/sockets after the deadline.
  • Regression tests cover runtime health loss and coordinated draining on completion, cancellation, and failure.
  • Sanitized request telemetry identifies attempted upstreams and connection/response timings; eviction and terminal-timeout accounting is available.
  • Live overload evidence demonstrates bounded failover, end-to-end 429/Retry-After propagation, and the client's AIMD/cooldown response.
  • Evidence distinguishes automated/loopback coverage from live-cluster and sealed-artifact acceptance.

Source References

Assessment against feat/slurm-execution on 2026-10-28:

Lingua principale
Python
Stelle
2.4k
Fork
241
Merge medio
1g 1h
PR unite (30g)
86

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA-NeMo/DataDesigner

Tutte le issue di NVIDIA-NeMo/DataDesigner

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.