[FEATURE] Persist runtime compile and autotune caches on the model cache
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes, pytorch
- Domain
- backend, infrastructure
Research direction
Start by reading the deployment builder in internal/controller/model_storage.go near the model-cache volume handling, then inspect the runtime backends and their BuildProbes methods to see how runtime-specific settings are declared. The issue calls for a new opt-in model-cache field, per-service mounts and runtime cache configuration, with deployment-builder tests for enabled, disabled and user-mount cases. Done means those behaviors are covered for each supported runtime.
Written by the indexing model from the issue text.
Description
Feature Description
An opt-in way to keep a runtime's compile and autotune caches on persistent storage, so a restarted or rescheduled inference pod skips most of its warmup.
Problem Statement
After the weights load, a vLLM start spends minutes building kernels and tuning, and writes the results inside the container, so every new container repeats the work. Measured on nvidia/GLM-5.3-Flash-NVFP4 (vLLM 0.28.1rc1, TP=4 on 4× B200), about 8 minutes passed between the end of the weight load and Application startup complete:
- CUDA graph capture: 166 s
- FlashInfer MoE autotuning (
trtllm_fp4_block_scale_moe): about 100 s, "Saved 44 configs to /root/.cache/vllm/flashinfer_autotune_cache/..." - TileLang and Triton kernel compiles
The caches it writes, all created after the container starts:
| Path | Written by |
|---|---|
/root/.cache/vllm |
vLLM, including the FlashInfer autotune cache |
/root/.cache/flashinfer |
FlashInfer JIT |
/root/.triton |
Triton |
/root/.tilelang |
TileLang |
/tmp/torchinductor_root |
TorchInductor (275 MB here) |
A startup-probe kill or a crash loop pays this every time, and so does every rollout.
The workaround today is subPath mounts on the volume the operator adds for spec.modelCache:
extraVolumeMounts:
- name: model-cache
mountPath: /root/.cache
subPath: _runtime-cache/glm-5-3-flash/dot-cache
- name: model-cache
mountPath: /root/.triton
subPath: _runtime-cache/glm-5-3-flash/triton
# ...and /root/.tilelang, /tmp/torchinductor_root
The mounts pass the file operations the caches need (see Additional Context), but:
- The volume name
model-cacheis hard-coded in the operator (internal/controller/model_storage.go:913) and isn't part of the API. A rename breaks every pod that relies on it. - Each user has to find every runtime's cache paths, which differ between runtimes and change between releases.
- A node-local volume (local-path) avoids the network but pins the service to one node through the PV's node affinity. The model cache, when it's ReadWriteMany, doesn't.
The ExtraVolumes field doc already names "a runtime-owned cache (e.g. a JIT kernel cache)" as a use case, so the need is anticipated; this asks for a first-class path.
Proposed Solution
An opt-in field on the model cache:
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
spec:
modelCache:
claimName: model-cache
runtimeCache: true # or a struct, e.g. {enabled: true, subPath: ...}
When set, the deployment builder:
- mounts a per-service directory from the model cache volume, for example
_runtime-cache/<namespace>/<name>/, read-write, beside the read-only/modelsmount; - points each runtime's caches into it, through env vars where the runtime has them (
VLLM_CACHE_ROOT,TRITON_CACHE_DIR,TORCHINDUCTOR_CACHE_DIR, FlashInfer's and TileLang's equivalents) or subPath mounts over the default paths where it doesn't; - leaves alone any path the user already mounts through
extraVolumeMounts, as the/dev/shmdefault does (#2111).
The per-runtime path list lives in the backend (for example a RuntimeCacheDirs() method beside BuildProbes), so vLLM, SGLang and TGI each declare their own.
Tests: a deployment-builder test that a service with runtimeCache gets the mount and env vars for its runtime, that one without it gets neither, and that a user mount at one of the paths wins.
Alternatives Considered
- An emptyDir: binds to no node and survives container restarts, but a new pod starts cold.
- local-path or hostPath: fast, but local-path pins the service to one node, and hostPath needs node setup and a privilege that Pod Security
baselineforbids. - A separate PVC per service: more claims to manage, and ReadWriteMany storage is needed anyway so the service can move between nodes.
Additional Context
- Tested on blobfuse2 (
azureblob-flex, file-cache mode): a pod with the vLLM image and a model-cache subPath passed write-then-rename,flock,mmap, anddlopenof a shared library from the mount, which are the operations Triton and TorchInductor use for their caches. - Blobfuse locks hold within one node only. Two replicas on different nodes compiling the same kernel at once could race; the caches write to temp files and rename them into place, which limits the damage, but this is worth noting in the docs.
- Cache keys include the GPU architecture and library versions, so nodes with different GPUs or a new image don't collide.
- Related: #2111 (the
/dev/shmdefault, the same "the operator knows the runtime's needs" pattern).
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day