[FEATURE] Persist runtime compile and autotune caches on the model cache

Open
#2,117 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes, pytorch

Research direction

Start by reading the deployment builder in internal/controller/model_storage.go near the model-cache volume handling, then inspect the runtime backends and their BuildProbes methods to see how runtime-specific settings are declared. The issue calls for a new opt-in model-cache field, per-service mounts and runtime cache configuration, with deployment-builder tests for enabled, disabled and user-mount cases. Done means those behaviors are covered for each supported runtime.

Written by the indexing model from the issue text.

Description

area/performance area/runtime component/api component/controller kind/feature priority/medium size/medium

Feature Description

An opt-in way to keep a runtime's compile and autotune caches on persistent storage, so a restarted or rescheduled inference pod skips most of its warmup.

Problem Statement

After the weights load, a vLLM start spends minutes building kernels and tuning, and writes the results inside the container, so every new container repeats the work. Measured on nvidia/GLM-5.3-Flash-NVFP4 (vLLM 0.28.1rc1, TP=4 on 4× B200), about 8 minutes passed between the end of the weight load and Application startup complete:

  • CUDA graph capture: 166 s
  • FlashInfer MoE autotuning (trtllm_fp4_block_scale_moe): about 100 s, "Saved 44 configs to /root/.cache/vllm/flashinfer_autotune_cache/..."
  • TileLang and Triton kernel compiles

The caches it writes, all created after the container starts:

Path Written by
/root/.cache/vllm vLLM, including the FlashInfer autotune cache
/root/.cache/flashinfer FlashInfer JIT
/root/.triton Triton
/root/.tilelang TileLang
/tmp/torchinductor_root TorchInductor (275 MB here)

A startup-probe kill or a crash loop pays this every time, and so does every rollout.

The workaround today is subPath mounts on the volume the operator adds for spec.modelCache:

extraVolumeMounts:
  - name: model-cache
    mountPath: /root/.cache
    subPath: _runtime-cache/glm-5-3-flash/dot-cache
  - name: model-cache
    mountPath: /root/.triton
    subPath: _runtime-cache/glm-5-3-flash/triton
  # ...and /root/.tilelang, /tmp/torchinductor_root

The mounts pass the file operations the caches need (see Additional Context), but:

  • The volume name model-cache is hard-coded in the operator (internal/controller/model_storage.go:913) and isn't part of the API. A rename breaks every pod that relies on it.
  • Each user has to find every runtime's cache paths, which differ between runtimes and change between releases.
  • A node-local volume (local-path) avoids the network but pins the service to one node through the PV's node affinity. The model cache, when it's ReadWriteMany, doesn't.

The ExtraVolumes field doc already names "a runtime-owned cache (e.g. a JIT kernel cache)" as a use case, so the need is anticipated; this asks for a first-class path.

Proposed Solution

An opt-in field on the model cache:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
spec:
  modelCache:
    claimName: model-cache
    runtimeCache: true     # or a struct, e.g. {enabled: true, subPath: ...}

When set, the deployment builder:

  • mounts a per-service directory from the model cache volume, for example _runtime-cache/<namespace>/<name>/, read-write, beside the read-only /models mount;
  • points each runtime's caches into it, through env vars where the runtime has them (VLLM_CACHE_ROOT, TRITON_CACHE_DIR, TORCHINDUCTOR_CACHE_DIR, FlashInfer's and TileLang's equivalents) or subPath mounts over the default paths where it doesn't;
  • leaves alone any path the user already mounts through extraVolumeMounts, as the /dev/shm default does (#2111).

The per-runtime path list lives in the backend (for example a RuntimeCacheDirs() method beside BuildProbes), so vLLM, SGLang and TGI each declare their own.

Tests: a deployment-builder test that a service with runtimeCache gets the mount and env vars for its runtime, that one without it gets neither, and that a user mount at one of the paths wins.

Alternatives Considered

  • An emptyDir: binds to no node and survives container restarts, but a new pod starts cold.
  • local-path or hostPath: fast, but local-path pins the service to one node, and hostPath needs node setup and a privilege that Pod Security baseline forbids.
  • A separate PVC per service: more claims to manage, and ReadWriteMany storage is needed anyway so the service can move between nodes.

Additional Context

  • Tested on blobfuse2 (azureblob-flex, file-cache mode): a pod with the vLLM image and a model-cache subPath passed write-then-rename, flock, mmap, and dlopen of a shared library from the mount, which are the operations Triton and TorchInductor use for their caches.
  • Blobfuse locks hold within one node only. Two replicas on different nodes compiling the same kernel at once could race; the caches write to temp files and rename them into place, which limits the damage, but this is worth noting in the docs.
  • Cache keys include the GPU architecture and library versions, so nodes with different GPUs or a new image don't collide.
  • Related: #2111 (the /dev/shm default, the same "the operator knows the runtime's needs" pattern).
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.