good first issuehelp wanted
Repository metrics
- Stars
- (8,345 個のスター)
- PR merge metrics
- (平均マージ 6d 13h) (30d で 204 merged PRs)
説明
Description:
follow the SSD offload guide in :
experimentation with vLLM and LMCache, we observed that the KV-Cache offloading to SSD works partially but lacks persistence across server restarts.
Steps to Reproduce:
- Configure vLLM server with SSD offloading for KV-Cache
- Run inference scripts with a specific prompt
- Observe 2.4GB
.ptfile generated in the SSD cache directory - Restart the vLLM server process
- Query the same prompt again
- the kv-cache regenerated in SSD
Expected Behavior:
The existing KV-Cache files should be reused, avoiding redundant computation and disk I/O.
Actual Behavior:
New identical KV-Cache files are regenerated instead of reusing the existing ones
This optimization would significantly improve latency for recurring queries in production scenarios. I would appreciate guidance on recommended methods to achieve proper KV-Cache persistence across server restarts in this workflow.