Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Idle unload never happens again if the client disconnects while a sleeping graph is waking up

Aperta
  • Qualcosa non va?
#4,603 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@atobiszei ci sta già lavorando.

Dal 25/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Describe the bug

With model groups and --idle_unload_timeout_seconds, a MediaPipe LLM graph is normally idle-unloaded after the timeout. If a client sends a request to a sleeping graph and disconnects while the graph is still waking up (reloading), the graph comes up as loaded but is never idle-unloaded again. Later normal requests don't fix this: the graph stays loaded until the server restarts.

With a 26B model this leaves ~16 GB of RAM allocated indefinitely. In our case it stayed loaded for ~4 hours with a 600 s timeout.

To Reproduce

Any small LLM works. The graph below is a standard HttpLLMCalculator graph with device: "GPU", max_num_seqs: 4, enable_prefix_caching: false, cache_size: 0.

config.json:

{"model_config_list":[{"config":{"name":"tiny","base_path":"/path/to/graph_dir","group_name":"g"}}]}

ovms --config_path config.json --rest_port 8011 --idle_unload_timeout_seconds 30 --metrics_enable

# graph starts SLEEPING; send a request and abort it during wake-up
timeout 0.5 curl -sN localhost:8011/v3/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"tiny","stream":true,"messages":[{"role":"user","content":"Hi"}],"max_tokens":10}'

# wait well beyond the timeout
sleep 90
curl -s localhost:8011/metrics | grep -E '^ovms_(graph_loaded|current_graphs|requests_accepted)'

Observed

  • Log: triggering lazy wake-up reload -> RELOADING -> AVAILABLE -> wake-up completed. After that there is no Idle unloading model group line, ever.
  • Metrics stay at ovms_graph_loaded{name="tiny"} 1, ovms_current_graphs{name="tiny"} 0, and all ovms_requests_accepted counters are 0 (the aborted request was never counted).
  • Sending further normal requests afterwards and waiting beyond the timeout: still not unloaded.

Expected behavior

The graph is idle-unloaded after --idle_unload_timeout_seconds like after any other request. An aborted request during wake-up should not keep the group "in use".

Control cases (all unload correctly after the timeout)

  • Normal unary request
  • Streaming request aborted by the client mid-generation (graph already loaded)

My guess is that the request aborted during wake-up is never counted in ovms_requests_accepted / finished, so the group's in-use reference is never released. I haven't checked this in the code.

Configuration

  • OVMS 2026.4.0.869b2186a (OpenVINO 2026.4.0-22959, GenAI 2026.4.0.0-3407), binary package for Ubuntu 24.04 running on Ubuntu 26.04
  • Intel Core Ultra X7 358H (Panther Lake), Arc B390 iGPU (device: "GPU"), 64 GB RAM
  • Models: gemma-4-26b-a4b-it-int4-ov (VLM continuous batching servable), reproduced with Qwen2.5-1.5B-Instruct-int4-ov

Workaround

An external watchdog reads /metrics and restarts the server if ovms_graph_loaded is 1, ovms_current_graphs is 0 and no ovms_requests_accepted counter has changed for 2× the idle timeout.

Lingua principale
C++
Stelle
932
Fork
278
Merge medio
3g 5h
PR unite (30g)
70

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di openvinotoolkit/model_server

Tutte le issue di openvinotoolkit/model_server

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.