Idle unload never happens again if the client disconnects while a sleeping graph is waking up
维护者通常 1 天内回复
@atobiszei 已经在做这个了。
开始于 2026年9月25日。
评估
这个 Issue 还没有评估数据。
描述
Describe the bug
With model groups and --idle_unload_timeout_seconds, a MediaPipe LLM graph is normally idle-unloaded after the timeout. If a client sends a request to a sleeping graph and disconnects while the graph is still waking up (reloading), the graph comes up as loaded but is never idle-unloaded again. Later normal requests don't fix this: the graph stays loaded until the server restarts.
With a 26B model this leaves ~16 GB of RAM allocated indefinitely. In our case it stayed loaded for ~4 hours with a 600 s timeout.
To Reproduce
Any small LLM works. The graph below is a standard HttpLLMCalculator graph with device: "GPU", max_num_seqs: 4, enable_prefix_caching: false, cache_size: 0.
config.json:
{"model_config_list":[{"config":{"name":"tiny","base_path":"/path/to/graph_dir","group_name":"g"}}]}
ovms --config_path config.json --rest_port 8011 --idle_unload_timeout_seconds 30 --metrics_enable
# graph starts SLEEPING; send a request and abort it during wake-up
timeout 0.5 curl -sN localhost:8011/v3/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"tiny","stream":true,"messages":[{"role":"user","content":"Hi"}],"max_tokens":10}'
# wait well beyond the timeout
sleep 90
curl -s localhost:8011/metrics | grep -E '^ovms_(graph_loaded|current_graphs|requests_accepted)'
Observed
- Log:
triggering lazy wake-up reload->RELOADING->AVAILABLE->wake-up completed. After that there is noIdle unloading model groupline, ever. - Metrics stay at
ovms_graph_loaded{name="tiny"} 1,ovms_current_graphs{name="tiny"} 0, and allovms_requests_acceptedcounters are 0 (the aborted request was never counted). - Sending further normal requests afterwards and waiting beyond the timeout: still not unloaded.
Expected behavior
The graph is idle-unloaded after --idle_unload_timeout_seconds like after any other request. An aborted request during wake-up should not keep the group "in use".
Control cases (all unload correctly after the timeout)
- Normal unary request
- Streaming request aborted by the client mid-generation (graph already loaded)
My guess is that the request aborted during wake-up is never counted in ovms_requests_accepted / finished, so the group's in-use reference is never released. I haven't checked this in the code.
Configuration
- OVMS 2026.4.0.869b2186a (OpenVINO 2026.4.0-22959, GenAI 2026.4.0.0-3407), binary package for Ubuntu 24.04 running on Ubuntu 26.04
- Intel Core Ultra X7 358H (Panther Lake), Arc B390 iGPU (
device: "GPU"), 64 GB RAM - Models: gemma-4-26b-a4b-it-int4-ov (VLM continuous batching servable), reproduced with Qwen2.5-1.5B-Instruct-int4-ov
Workaround
An external watchdog reads /metrics and restarts the server if ovms_graph_loaded is 1, ovms_current_graphs is 0 and no ovms_requests_accepted counter has changed for 2× the idle timeout.
- 主要语言
- C++
- 星标
- 932
- 派生
- 278
- 平均合并
- 3 天 4 小时
- 30 天内合并 PR
- 68
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
openvinotoolkit/model_server 的其他 Issue
-
难度 3/5 1-2 天 新手友好度 58/100
openvinotoolkit/model_server#4613 · 已指派 1 人 ·
维护者通常 1 天内回复
-
enhancement
难度 3/5 1-2 天 新手友好度 68/100
openvinotoolkit/model_server#4609 · 1 条评论 ·
维护者通常 1 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 45/100
openvinotoolkit/model_server#4599 · 4 条评论 ·
维护者通常 1 天内回复
-
难度 3/5 1-2 天 新手友好度 55/100
openvinotoolkit/model_server#4586 · 5 条评论 ·
维护者通常 1 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 52/100
openvinotoolkit/model_server#4580 · 8 条评论 ·
维护者通常 1 天内回复
查看 openvinotoolkit/model_server 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 75/100
mpfaffenberger/privateer_reimagined#658 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
microsoft/onnxruntime#33018 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 82/100
AXERA-TECH/ax-llm#81 ·
-
enhancement
难度 2/5 半天 新手友好度 78/100
ros-industrial/ros2_canopen#448 ·
-
难度 1/5 1 小时以内 新手友好度 78/100
维护者通常 1 天内回复