Synchronous model loads run inside the shared workflows thread pool, stalling all concurrent workflow runs (latency collapse with idle GPU/CPU)

Aperta
  • Qualcosa non va?
#2,448 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
python
Ambito
backend, machine-learning, performance

Direzione di ricerca

Inizia tracciando le chiamate a shared_thread_pool_executor e ExecutionEngine.init in inference/core/interfaces/http/http_api.py, quindi analizza il locking e l’eviction della cache in inference/core/managers/decorators/fixed_size_cache.py. Riproduci l’esaurimento del pool con caricamenti sincroni dei modelli e rivedi i test di workflow esistenti. Il lavoro è completato quando i caricamenti non occupano più lo shared step pool e sono presenti test di copertura per l’esecuzione concorrente dei workflow e per il comportamento di caricamento richiesto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

On a GPU inference server (roboflow-inference-server-gpu, build of 2026-06-05 / v1.3.0 era) serving multi-model Workflows under sustained production traffic, we observed a latency collapse mode where p50 latency of all workflow endpoints rises 3–10x while accelerator duty cycle stays ≤13% and CPU stays under 25% of limit. Raw /infer/* endpoints on the same pods remain fast. Root cause analysis points at a structural issue: synchronous model loads (3–19s each) execute inside the shared workflows ThreadPoolExecutor, starving step execution for every concurrent workflow run in the process.

Environment

  • gpu_http:app via uvicorn, 1 worker, defaults: HTTP_API_SHARED_WORKFLOWS_THREAD_POOL_ENABLED=True (16 workers), WORKFLOWS_MAX_CONCURRENT_STEPS=4, MAX_ACTIVE_MODELS=20, MEMORY_FREE_THRESHOLD=0.20
  • Workload: ~6 workflow variants chaining 3–5 models each; working set ~12 model/version combos (~600MB VRAM each) on NVIDIA L4
  • Horizontal autoscaling on request rate; replica churn redistributes traffic across pods regularly

Observed

  • Model loaded: ... load_time= events: p50 ≈ 5s, p90 ≈ 11s, occurring continuously in steady state (~200–700/hr fleet-wide) because the model working set + MEMORY_FREE_THRESHOLD VRAM guard keeps the WithFixedSizeCache manager evicting (reason=memory_pressure) and re-loading
  • When autoscaler churn raises per-pod cache-miss rates, workflow p50/p90 explodes (0.8s → 4–8s p90 across all workflow endpoints uniformly) while GPU duty cycle ≤13% and CPU well under limit — classic heavy-tailed-service queueing on a software bottleneck (E[S²] dominated by inline loads), not a hardware one
  • /infer/* on the same pods degrades only mildly (~2x p90 at worst) — consistent with it not using the shared workflows pool, only the model-manager lock

Code path

  • inference/core/interfaces/http/http_api.py (~L1193): one process-global self.shared_thread_pool_executor = ThreadPoolExecutor(max_workers=16) is created and passed to every ExecutionEngine.init(...) (~L1293)
  • Model blocks execute as steps in that pool; a cold model triggers add_model → download + deserialize + VRAM upload (3–19s) inside a step worker, while also holding the WithFixedSizeCache queue lock for eviction bookkeeping (inference/core/managers/decorators/fixed_size_cache.py), so concurrent runs both lose a pool worker and contend on the lock
  • Under churn, several in-flight loads consume a large fraction of the 16 workers; queued steps back up; every workflow run slows; in-flight requests then pile up to whatever the ASGI concurrency limit is, producing 503 load shedding while the GPU is idle

Asks

  1. Isolate model loading from step execution — run loads on a dedicated loader executor (or async path) so a cold model only blocks requests that need that model, never the shared step pool
  2. First-class model pre-warming: a way to declare the deployment's model set to be loaded at startup (and pinned), with readiness gated on warm-up, so steady-state inline loads approach zero (the _pinned_models machinery in WithFixedSizeCache looks close but isn't exposed/documented for this)
  3. Hysteresis / actively-used protection on memory_pressure evictions — under the VRAM threshold, LRU eviction of models that are in active rotation creates a permanent evict↔reload cycle
  4. Observability: counters/histograms for model load duration, cache miss rate, and shared-pool queue depth would have made this diagnosable from metrics instead of log archaeology

Happy to provide more detail on the analysis.

Lingua principale
Python
Stelle
2.5k
Fork
340
Merge medio
2g 11h
PR unite (30g)
92

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di roboflow/inference

Tutte le issue di roboflow/inference

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.