[FEATURE] multiNode: stage weights per member instead of requiring a pre-existing claim per node

Open
#1,752 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
go, kubernetes

Research direction

Start with internal/controller/multinode.go, model_storage.go, and model_prefetch.go to trace how member claims, model downloads, and prefetch jobs work; then compare the storage behavior with docs/site/guides/multi-node-inference.md. Implement per-member staging while preserving members[i].modelCache.claimName as an override, and update the API doc comments and guide. Done means each member can receive weights without a pre-existing claim, and group readiness waits for staging across members.

Written by the indexing model from the issue text.

Description

enhancement

Feature Description

Stage a Model's weights into every member of a spec.multiNode group, so a group no longer requires a pre-existing, user-owned claim per node.

Problem Statement

#1746 ships members[i].modelCache.claimName: each member mounts its own claim, and the claims must already hold the weights at the same relative path. That was the right v1 cut, but it moves the hardest part onto the user. On the two-Spark ring it meant hand-staging 135 GB twice, and the claims came out with different layouts (a Hugging Face cache snapshot on one node, a plain directory on the other), which needed a root-owned relative symlink before the group could start (see the Storage section of docs/site/guides/multi-node-inference.md). Nothing in the operator's download and prefetch machinery (internal/controller/model_storage.go, model_prefetch.go) knows about members: modelNeedsCachePVC and ensureModelCachePVC assume one per-service claim.

Proposed Solution

  • In internal/controller/multinode.go, when a member has no modelCache.claimName, create a per-member claim named <isvc>-mn-<rank>-model-cache with the same size and class rules the service claim uses, bound to the member's node (a local-path or the configured class, with node affinity through a scheduled staging pod rather than a PV by hand).
  • Run the existing model-downloader init container on every member pod (it already renders per pod from buildModelStorageConfig); the head must not wait on its own download only, so the group readiness stays "every member Running and rank 0 Ready", which already covers members still staging.
  • The prefetch Job (model_prefetch.go) gains a per-member mode that runs one Job per member node so weights are warm before the group is created; reuse the reviseFromBranch-style pattern of naming Jobs by rank.
  • Keep members[i].modelCache.claimName as the explicit override.

Files: internal/controller/multinode.go, internal/controller/model_storage.go, internal/controller/model_prefetch.go, api/v1alpha1/inferenceservice_types.go (doc comments only), docs/site/guides/multi-node-inference.md.

Additional Context

Parent: #1423. Shipped v1: #1746. Layout lesson: the Hugging Face cache trap documented in the guide is the reason the relative path must match across members.

Priority

  • Medium - Nice to have

Willingness to Contribute

  • Yes, I can submit a PR
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.