[FEATURE] Keep staged weights resident in the node-local cache, not just present in the claim
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- infrastructure
Research direction
Start by reading the model cache spec, prefetch status handling, claim resolution, and serving pod placement logic; the issue names no source files or tests. Trace how a service selects nodes and mounts its claim, then decide how an operator-managed process could keep the staged files warm on those nodes. Done means an opt-in setting keeps the recorded files resident through redeploys; the issue proposes a DaemonSet or equivalent but leaves the design open.
Written by the indexing model from the issue text.
Description
Feature Description
Let LLMKube keep a model's weights resident in the node-local cache in front of a network-backed model cache claim, not just present in the claim.
Prefetch (#2136 / #2138) answers "are the weights in the claim?". On a FUSE-backed claim there is a second question it does not reach: "are the weights on the node that will serve them?" A CSI driver such as blob.csi.azure.com keeps a local file cache per node, and that cache lives exactly as long as the mount, not as long as the pod.
Problem Statement
As an operator serving large models from a shared RWX claim, I want a redeploy not to cost a full re-read of the weights over the network, so that a routine config change is not a long outage.
kubelet calls NodeUnstageVolume once the last pod on a node stops using a claim. The driver then unmounts and clears its local cache directory. The next pod start re-reads every byte over the network.
LLMKube makes this deterministic rather than occasional: it generates strategy: Recreate for serving Deployments — correctly, because on a node whose GPUs are fully allocated there is nothing to surge into and RollingUpdate could never complete. So every redeploy drops that node to zero consumers.
Measured on one 8-GPU node, a single NVFP4 checkpoint is 190 GiB across 42 files; the whole claim was 216 GiB across 65 files. Re-reading that on every tuning change is the cost today.
This also gets sharper with spec.offline: true from #2138: the serving pod has no upstream fallback at all, so a cold local cache sits on the critical path of every start with nothing behind it.
Proposed Solution
Opt-in on the cache spec, since only a FUSE/CSI-backed claim has this property:
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
name: example
spec:
modelCache:
claimName: model-cache
prefetch: true
# Keep this Model's staged files resident in the node-local cache on the
# nodes this service is placed on.
keepResident: true
LLMKube is the component that can actually do this well, because it already knows the three things required:
- which files —
status.prefetch[].filesalready records exactly the set the Job staged andmodel-verifychecks. - which claim —
modelCache.claimNameresolution already exists on both kinds. - which nodes — it owns the serving pods' placement.
A reasonable shape is an operator-managed DaemonSet (or equivalent) scoped to the nodes that service is placed on, which holds the claim mounted and touches the recorded file set periodically.
Two mechanics worth knowing for whoever picks this up:
- With
file_cache, opening a file downloads the whole file. One byte per file is a complete warm; reading the rest only pulls it back out through FUSE. A pass over files already cached costs metadata, not transfer. - Entries leave the cache some interval after close (
--file-cache-timeout-in-seconds), so a periodic re-open inside that window is what keeps them resident.
Alternatives Considered
- A CronJob. Self-defeating: between runs its pod is the only consumer, so the unmount on exit clears the cache the run just filled. It only works if something else holds the mount. It is also one pod, where the cache is per node.
- Raise
--file-cache-timeout-in-seconds. Governs eviction while mounted. Teardown is not eviction; the directory goes regardless of the timeout. - Pin the driver's
--tmp-pathto a stable host path. The driver adds--empty-dir-checkto every mount, so a cache directory that survived would fail the next mount. - Serve from node-local storage and treat the network claim as origin. Works, but gives up the shared claim and costs a per-node copy plus a staging step — much of what prefetch exists to provide.
- Never let the consumer count reach zero. What we do today (below). It works, but every operator has to rediscover the mechanism and hand-maintain a DaemonSet that names a claim, while LLMKube already holds the file list.
Additional Context
- Related issues: #2136, #2138 (
spec.offlinemakes a cold cache unrecoverable at serve time rather than merely slow) - Workaround in use: a DaemonSet in our platform repo that mounts the claim read-only on the serving nodes and re-opens every file hourly. It holds the mount open, which is what makes the warm durable. Roughly:
while :; do
find /model-cache -type f -exec head -c1 {} + >/dev/null
sleep 3600
done
Before that, the workaround was rolling one model config at a time so that one pod always held the mount.
Priority
- Critical - Blocking my use case
- High - Would significantly improve my workflow
- Medium - Nice to have
- Low - Minor improvement
Medium because we have a working local answer. It is filed because the operator is the right owner: it knows the file set and the placement, and every user of a FUSE-backed cache otherwise rediscovers this the hard way.
Willingness to Contribute
- Yes, I can submit a PR
- Yes, I can help test
- No, but I can provide feedback
- No, just requesting
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
bug ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
RedHatInsights/insights-ingress-go#946 · 4 comments ·
Maintainers usually reply within 1 day