[FEATURE] Keep staged weights resident in the node-local cache, not just present in the claim

Open
#2,145 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes

Research direction

Start by reading the model cache spec, prefetch status handling, claim resolution, and serving pod placement logic; the issue names no source files or tests. Trace how a service selects nodes and mounts its claim, then decide how an operator-managed process could keep the staged files warm on those nodes. Done means an opt-in setting keeps the recorded files resident through redeploys; the issue proposes a DaemonSet or equivalent but leaves the design open.

Written by the indexing model from the issue text.

Description

kind/feature priority/low

Feature Description

Let LLMKube keep a model's weights resident in the node-local cache in front of a network-backed model cache claim, not just present in the claim.

Prefetch (#2136 / #2138) answers "are the weights in the claim?". On a FUSE-backed claim there is a second question it does not reach: "are the weights on the node that will serve them?" A CSI driver such as blob.csi.azure.com keeps a local file cache per node, and that cache lives exactly as long as the mount, not as long as the pod.

Problem Statement

As an operator serving large models from a shared RWX claim, I want a redeploy not to cost a full re-read of the weights over the network, so that a routine config change is not a long outage.

kubelet calls NodeUnstageVolume once the last pod on a node stops using a claim. The driver then unmounts and clears its local cache directory. The next pod start re-reads every byte over the network.

LLMKube makes this deterministic rather than occasional: it generates strategy: Recreate for serving Deployments — correctly, because on a node whose GPUs are fully allocated there is nothing to surge into and RollingUpdate could never complete. So every redeploy drops that node to zero consumers.

Measured on one 8-GPU node, a single NVFP4 checkpoint is 190 GiB across 42 files; the whole claim was 216 GiB across 65 files. Re-reading that on every tuning change is the cost today.

This also gets sharper with spec.offline: true from #2138: the serving pod has no upstream fallback at all, so a cold local cache sits on the critical path of every start with nothing behind it.

Proposed Solution

Opt-in on the cache spec, since only a FUSE/CSI-backed claim has this property:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: example
spec:
  modelCache:
    claimName: model-cache
    prefetch: true
    # Keep this Model's staged files resident in the node-local cache on the
    # nodes this service is placed on.
    keepResident: true

LLMKube is the component that can actually do this well, because it already knows the three things required:

  • which files — status.prefetch[].files already records exactly the set the Job staged and model-verify checks.
  • which claim — modelCache.claimName resolution already exists on both kinds.
  • which nodes — it owns the serving pods' placement.

A reasonable shape is an operator-managed DaemonSet (or equivalent) scoped to the nodes that service is placed on, which holds the claim mounted and touches the recorded file set periodically.

Two mechanics worth knowing for whoever picks this up:

  • With file_cache, opening a file downloads the whole file. One byte per file is a complete warm; reading the rest only pulls it back out through FUSE. A pass over files already cached costs metadata, not transfer.
  • Entries leave the cache some interval after close (--file-cache-timeout-in-seconds), so a periodic re-open inside that window is what keeps them resident.

Alternatives Considered

  • A CronJob. Self-defeating: between runs its pod is the only consumer, so the unmount on exit clears the cache the run just filled. It only works if something else holds the mount. It is also one pod, where the cache is per node.
  • Raise --file-cache-timeout-in-seconds. Governs eviction while mounted. Teardown is not eviction; the directory goes regardless of the timeout.
  • Pin the driver's --tmp-path to a stable host path. The driver adds --empty-dir-check to every mount, so a cache directory that survived would fail the next mount.
  • Serve from node-local storage and treat the network claim as origin. Works, but gives up the shared claim and costs a per-node copy plus a staging step — much of what prefetch exists to provide.
  • Never let the consumer count reach zero. What we do today (below). It works, but every operator has to rediscover the mechanism and hand-maintain a DaemonSet that names a claim, while LLMKube already holds the file list.

Additional Context

  • Related issues: #2136, #2138 (spec.offline makes a cold cache unrecoverable at serve time rather than merely slow)
  • Workaround in use: a DaemonSet in our platform repo that mounts the claim read-only on the serving nodes and re-opens every file hourly. It holds the mount open, which is what makes the warm durable. Roughly:
while :; do
  find /model-cache -type f -exec head -c1 {} + >/dev/null
  sleep 3600
done

Before that, the workaround was rolling one model config at a time so that one pod always held the mount.

Priority

  • Critical - Blocking my use case
  • High - Would significantly improve my workflow
  • Medium - Nice to have
  • Low - Minor improvement

Medium because we have a working local answer. It is filed because the operator is the right owner: it knows the file set and the placement, and every user of a FUSE-backed cache otherwise rediscovers this the hard way.

Willingness to Contribute

  • Yes, I can submit a PR
  • Yes, I can help test
  • No, but I can provide feedback
  • No, just requesting
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.