Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Aggressive retry mechanism causing VLLM shared memory exhaustion during streaming

Aperta
  • Qualcosa non va?
#279 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python
Ambito
api, backend

Direzione di ricerca

Start by locating the Python client implementation containing DEFAULT_MAX_RETRIES, INITIAL_RETRY_DELAY, and _should_retry, then trace how options.stream is handled. Reproduce or test a streaming request with retries enabled and determine the intended configuration behavior. Done means streaming retries no longer create redundant requests or exhaust VLLM shared memory, with coverage for the selected retry policy.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Issue Description

When using the LlamaStack Python client with VLLM backend, the retry mechanism during streaming responses can lead to VLLM shared memory exhaustion.

System Architecture
VLLM <= LlamaStack <= Client Application
Observed Behavior
  1. Initial streaming request is made and starts receiving chunks
  2. While waiting for VLLM response chunks, LlamaStack client automatically retries the request
  3. Multiple identical requests accumulate, eventually causing VLLM to run out of shared memory
Logs Evidence

First attempt:

DEBUG 10-10 05:27:15 [client.py:193] Waiting for output from MQLLMEngine.
DEBUG 10-10 05:27:16 [metrics.py:386] Avg prompt throughput: 0.0 tokens/s

Second attempt (automatic retry):

INFO 10-10 05:27:20 [logger.py:41] Received request chatcmpl-912e2a2b68f44d46a991b3b37cf7895a

Third attempt (another retry):

INFO 10-10 05:27:51 [logger.py:41] Received request chatcmpl-ea4023145df84ba183683a38983e5977

Final error:

ERROR 10-10 05:28:03 [serving_chat.py:932] vllm.engine.multiprocessing.MQEngineDeadError: Engine loop is not running. Inspect the stacktrace to find the original error: OutOfResources(98304, 65536, 'shared memory').
Current Implementation

The client currently has the following retry behavior:

  1. Default configuration:
DEFAULT_MAX_RETRIES = 2
MAX_RETRY_DELAY = 8  # seconds
INITIAL_RETRY_DELAY = 0.5  # seconds
  1. Retry conditions (from _should_retry):
# Retry on request timeouts (408)
# Retry on lock timeouts (409)
# Retry on rate limits (429)
# Retry on server errors (502, 503, 504)
Impact
  • VLLM crashes due to shared memory exhaustion
  • Multiple redundant requests consume resources
  • Potential cascading failures in the system
Suggested Solutions
  1. Add streaming-specific retry configuration:
class StreamingConfig:
    max_retries: int = 0
    retry_on_timeout: bool = False
    retry_on_server_error: bool = False
  1. Add a header to indicate streaming requests that should bypass retries:
X-LlamaStack-Streaming: true
  1. Or simply detect streaming requests and disable retries automatically:
if options.stream:
    options.max_retries = 0

The third option might be the safest as it would prevent any potential memory exhaustion issues by default while still allowing users to explicitly enable retries if needed.

Questions
  1. Is there a way to configure retry behavior specifically for streaming requests in the current version?
  2. Are there plans to add more granular retry control?
  3. Is there a recommended way to handle streaming retries differently from regular requests?
Environment
  • LlamaStack Python Client (latest version)
  • VLLM backend
  • Python 3.x
Lingua principale
Python
Stelle
198
Fork
102
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ogx-ai/ogx-client-python

Tutte le issue di ogx-ai/ogx-client-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.