Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Aggressive retry mechanism causing VLLM shared memory exhaustion during streaming

オープン
  • 何か問題がありますか？
#279 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
python
領域
api, backend

調査の方向性

Start by locating the Python client implementation containing DEFAULT_MAX_RETRIES, INITIAL_RETRY_DELAY, and _should_retry, then trace how options.stream is handled. Reproduce or test a streaming request with retries enabled and determine the intended configuration behavior. Done means streaming retries no longer create redundant requests or exhaust VLLM shared memory, with coverage for the selected retry policy.

索引モデルが issue の本文から書いたものです。

説明

Issue Description

When using the LlamaStack Python client with VLLM backend, the retry mechanism during streaming responses can lead to VLLM shared memory exhaustion.

System Architecture
VLLM <= LlamaStack <= Client Application
Observed Behavior
  1. Initial streaming request is made and starts receiving chunks
  2. While waiting for VLLM response chunks, LlamaStack client automatically retries the request
  3. Multiple identical requests accumulate, eventually causing VLLM to run out of shared memory
Logs Evidence

First attempt:

DEBUG 10-10 05:27:15 [client.py:193] Waiting for output from MQLLMEngine.
DEBUG 10-10 05:27:16 [metrics.py:386] Avg prompt throughput: 0.0 tokens/s

Second attempt (automatic retry):

INFO 10-10 05:27:20 [logger.py:41] Received request chatcmpl-912e2a2b68f44d46a991b3b37cf7895a

Third attempt (another retry):

INFO 10-10 05:27:51 [logger.py:41] Received request chatcmpl-ea4023145df84ba183683a38983e5977

Final error:

ERROR 10-10 05:28:03 [serving_chat.py:932] vllm.engine.multiprocessing.MQEngineDeadError: Engine loop is not running. Inspect the stacktrace to find the original error: OutOfResources(98304, 65536, 'shared memory').
Current Implementation

The client currently has the following retry behavior:

  1. Default configuration:
DEFAULT_MAX_RETRIES = 2
MAX_RETRY_DELAY = 8  # seconds
INITIAL_RETRY_DELAY = 0.5  # seconds
  1. Retry conditions (from _should_retry):
# Retry on request timeouts (408)
# Retry on lock timeouts (409)
# Retry on rate limits (429)
# Retry on server errors (502, 503, 504)
Impact
  • VLLM crashes due to shared memory exhaustion
  • Multiple redundant requests consume resources
  • Potential cascading failures in the system
Suggested Solutions
  1. Add streaming-specific retry configuration:
class StreamingConfig:
    max_retries: int = 0
    retry_on_timeout: bool = False
    retry_on_server_error: bool = False
  1. Add a header to indicate streaming requests that should bypass retries:
X-LlamaStack-Streaming: true
  1. Or simply detect streaming requests and disable retries automatically:
if options.stream:
    options.max_retries = 0

The third option might be the safest as it would prevent any potential memory exhaustion issues by default while still allowing users to explicitly enable retries if needed.

Questions
  1. Is there a way to configure retry behavior specifically for streaming requests in the current version?
  2. Are there plans to add more granular retry control?
  3. Is there a recommended way to handle streaming retries differently from regular requests?
Environment
  • LlamaStack Python Client (latest version)
  • VLLM backend
  • Python 3.x
主要言語
Python
スター
198
フォーク
102
30日以内にマージされた PR はありません

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ogx-ai/ogx-client-python のほかの issue

ogx-ai/ogx-client-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。