Harden Slurm inference routing, backpressure, and failover

Priority Level

High

Task Summary

Replace the current minimal Slurm logical-endpoint proxy behavior with production-grade inference routing that provides bounded failover, queue-aware backpressure, streaming responses, connection reuse, aggregate readiness, and actionable upstream telemetry.

The proxy must remain transparent to Data Designer's request-admission controller: after bounded local failover is exhausted, an HTTP 429 response and its Retry-After header must reach the client so its AIMD controller can reduce concurrency.

Technical Details & Implementation Plan

Queue admission

• Continue sampling each vLLM backend's waiting-request metric every 100 milliseconds.
• Treat samples older than one second as stale and fail open when metrics are unavailable or stale.
• Keep the default maximum waiting-request threshold at 128; reject requests when the observed queue depth is greater than 128.
• Return HTTP 429 with Retry-After: 1 by default.
• Preserve the existing typed override when explicitly configured.

Load balancing and bounded failover

• Use least-connections backend selection with deterministic tie-breaking.
• Attempt at most three distinct backends for one client request.
• Permit bounded failover for POST and other non-idempotent inference requests.
• Retry another backend for HTTP 429, 500, 502, 503, or 504, connection failures, and timeouts before a response begins.
• Never retry after a response body has begun streaming.
• Preserve the final upstream response after the attempt limit is exhausted, especially the final 429 body and Retry-After header.

HTTP and connection management

• Keep client-facing HTTP/1.1 connections open when permitted by the protocol.
• Maintain a shared upstream connection pool for each model deployment.
• Allow up to 256 idle upstream keepalive connections per endpoint process.
• Reuse upstream connections across requests and client sessions with a 65-second idle timeout.
• Enable TCP keepalive and TCP_NODELAY.
• Validate pooled connections before reuse and discard stale or failed connections.
• Apply bounded per-backend connection limits so one replica cannot consume the entire pool.
• Do not count idle pooled connections as active requests for least-connections selection.
• Use a five-second upstream connection timeout and a 3,600-second read/write timeout.
• Stream response bodies without buffering the complete response in proxy memory.
• Support request bodies up to 100 MiB and strip hop-by-hop headers in both directions.
• Drain and close connection pools cleanly when an attempt completes, fails, or is cancelled.
• Do not send Connection: close except when required by an error or protocol condition.

Placement and lifecycle

• Run one CPU-only logical endpoint per model deployment on the deployment's client/head node.
• Do not consume a GPU for proxy, health, or telemetry processes.
• Start the client only after all required inference backends and logical endpoints are ready.
• Stop the endpoint process when its managed attempt terminates, is cancelled, or fails.
• Do not require users to build or register a separate proxy image or install a system proxy.

Readiness and observability

• Expose aggregate readiness that succeeds only when every required backend is healthy.
• Continue checking backend health after startup so managed state detects runtime service loss.
• Record selected upstream, attempt count, upstream status sequence, connection and response latency, final status, and bounded-failover usage without recording request or response content.
• Expose counters for requests, active requests, connection reuse, connection eviction/failure, local retries by reason, final 429 responses, final 5xx responses, and timeouts.
• Keep secrets, prompts, generated content, and authorization headers out of logs.

Acceptance Criteria

• Healthy replicas receive concurrent requests using least-connections.
• Sequential requests reuse a healthy upstream connection.
• Concurrent requests reuse the pool without exceeding configured limits.
• A stale pooled connection is discarded and retried within the three-attempt budget.
• Idle pooled connections do not distort least-connections selection.
• A 429 from one backend is retried against another eligible backend.
• No request makes more than three upstream attempts.
• If every attempted backend returns 429, the final 429 and Retry-After reach Data Designer's AIMD controller.
• HTTP 500, 502, 503, 504, connection failures, and pre-response timeouts trigger bounded failover.
• A response that has begun streaming is never replayed.
• Long-running streamed responses are forwarded incrementally without full-response buffering.
• Startup readiness fails unless every required backend is healthy.
• Runtime loss of a required backend is reflected in managed readiness and attempt state.
• Proxy, health, and telemetry steps request no GPU resources.
• Shutdown drains active responses for a bounded period and then closes remaining sockets.
• Users do not need to install a system proxy or provide an additional proxy image.
• Structured logs and counters identify routing and overload behavior without recording sensitive payloads.

Required Tests

• Unit tests for least-connections selection, deterministic tie-breaking, connection pooling, and the three-attempt limit.
• Unit tests for every retryable status and failure class.
• Unit tests proving final 429 and Retry-After preservation.
• Unit tests proving streaming responses are not replayed.
• Unit tests for stale or unavailable queue metrics failing open and the default waiting-request threshold of 128.
• Integration tests with eight mock inference replicas covering normal balancing, partial and total overload, mixed responses, connection failures, timeouts, 5xx failover, streaming, connection reuse, and backend loss after startup.
• Rendered Slurm-script tests proving CPU-only placement and lifecycle cleanup.
• End-to-end cluster validation with exact output counts, failure/retry counts, elapsed time, records per minute, and proxy CPU/memory telemetry.

Non-goals

• External ingress, TLS termination, authentication, or multi-tenant routing.
• Changing Data Designer's client-side AIMD algorithm.
• Retrying requests after response streaming begins.
• Adding a user-managed proxy image.
• Changing model placement or GPU topology.
• Adding sandbox-service routing.

Investigation / Context

The current endpoint supplies basic least-active routing and retries overload responses across replicas. However, it opens a new upstream connection per request, buffers complete responses, has a shorter timeout, retries overload against every backend, and does not immediately fail over for upstream errors, timeouts, or 5xx responses. These differences can delay client-side concurrency reduction and create a proxy bottleneck during larger generation runs.

Dependencies

None.