[FEATURE] multiNode: stage weights per member instead of requiring a pre-existing claim per node
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
Research direction
Start with internal/controller/multinode.go, model_storage.go, and model_prefetch.go to trace how member claims, model downloads, and prefetch jobs work; then compare the storage behavior with docs/site/guides/multi-node-inference.md. Implement per-member staging while preserving members[i].modelCache.claimName as an override, and update the API doc comments and guide. Done means each member can receive weights without a pre-existing claim, and group readiness waits for staging across members.
Written by the indexing model from the issue text.
Description
Feature Description
Stage a Model's weights into every member of a spec.multiNode group, so a group no longer requires a pre-existing, user-owned claim per node.
Problem Statement
#1746 ships members[i].modelCache.claimName: each member mounts its own claim, and the claims must already hold the weights at the same relative path. That was the right v1 cut, but it moves the hardest part onto the user. On the two-Spark ring it meant hand-staging 135 GB twice, and the claims came out with different layouts (a Hugging Face cache snapshot on one node, a plain directory on the other), which needed a root-owned relative symlink before the group could start (see the Storage section of docs/site/guides/multi-node-inference.md). Nothing in the operator's download and prefetch machinery (internal/controller/model_storage.go, model_prefetch.go) knows about members: modelNeedsCachePVC and ensureModelCachePVC assume one per-service claim.
Proposed Solution
- In
internal/controller/multinode.go, when a member has nomodelCache.claimName, create a per-member claim named<isvc>-mn-<rank>-model-cachewith the same size and class rules the service claim uses, bound to the member's node (alocal-pathor the configured class, with node affinity through a scheduled staging pod rather than a PV by hand). - Run the existing model-downloader init container on every member pod (it already renders per pod from
buildModelStorageConfig); the head must not wait on its own download only, so the group readiness stays "every member Running and rank 0 Ready", which already covers members still staging. - The prefetch Job (
model_prefetch.go) gains a per-member mode that runs one Job per member node so weights are warm before the group is created; reuse thereviseFromBranch-style pattern of naming Jobs by rank. - Keep
members[i].modelCache.claimNameas the explicit override.
Files: internal/controller/multinode.go, internal/controller/model_storage.go, internal/controller/model_prefetch.go, api/v1alpha1/inferenceservice_types.go (doc comments only), docs/site/guides/multi-node-inference.md.
Additional Context
Parent: #1423. Shipped v1: #1746. Layout lesson: the Hugging Face cache trap documented in the guide is the reason the relative path must match across members.
Priority
- Medium - Nice to have
Willingness to Contribute
- Yes, I can submit a PR
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day