[FEATURE] multiNode: operator-owned page-cache discipline and MemFree guard for unified-memory (GB10) members
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
Research direction
Start with api/v1alpha1/inferenceservice_types.go and internal/controller/multinode.go to trace MultiNodeSpec and member pod construction. Read internal/controller/multinode_test.go and inspect config/samples/gb10-memory-guard.yaml for existing behavior. Done means the operator can configure pre-boot cache drops, serving memory guards, and sysctls for each member, with the proposed metrics and tests.
Written by the indexing model from the issue text.
Description
Feature Description
When a multiNode vLLM group pins its KV cache with --kv-cache-memory-bytes, vLLM skips memory
profiling entirely (its log: "reserved N GiB memory for KV Cache as specified by kv_cache_memory_bytes
config and skipped memory profiling. This does not respect the gpu_memory_utilization config"). On a
DGX Spark host RAM is GPU memory, so nothing then protects the group from a dirty page cache at boot
or from the page cache regrowing while serving (Engram row reads on DeepSeek-V4.1 do exactly that).
Today that protection is a standalone privileged DaemonSet sample with its own ServiceAccount and Role
(config/samples/gb10-memory-guard.yaml). The operator should own it.
Problem Statement
As a maintainer running DeepSeek-V4.1-Flash EXL3 on a three-Spark ring at 512K context with vision,
I want the operator to guarantee the pre-boot cache drop and the serving-phase memory guard for a
pinned multiNode group, so a boot cannot start on a dirty cache and a member cannot starve the GPU
allocator, without applying a second manifest with separate RBAC.
Measured on 2026-09-20 (three GB10 members, 512K, vision, KV pin 4.2 GiB per rank):
| condition during a 450,977-token prefill | lowest MemFree on a rank |
|---|---|
| no flusher | 1.0 GiB |
| a 15 s flusher loop | 2.7 GiB |
the guard DaemonSet (2 s loop) plus vm.min_free_kbytes=1048576, vm.watermark_scale_factor=200 |
no drop needed, ~10 GiB after |
The nodes shipped at vm.min_free_kbytes 44 MiB and watermark_scale_factor 10. A fresh 460 GB
checkpoint stage also leaves ~10 GiB of page cache resident even with the group suspended.
Proposed Solution
A memory block on spec.multiNode that the controller implements for every member:
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
spec:
multiNode:
memory:
dropPageCacheBeforeBoot: true # privileged step on the member's node right before its pod starts
flushBelowMemFreeGiB: 4 # serving-phase flusher (drop caches when MemFree falls under this)
suspendBelowMemFreeGiB: 1.5 # guard: suspend the group once if MemFree stays under this
sysctls: # applied on the member node, logged before/after
vm.min_free_kbytes: "1048576"
vm.watermark_scale_factor: "200"
Implementation sketch:
api/v1alpha1/inferenceservice_types.go:MultiNodeSpec(line 271) gainsMemory *MultiNodeMemory;
make manifeststhenmake chart-crds.internal/controller/multinode.go:constructMemberPodsadds a privileged init container that drops
caches whendropPageCacheBeforeBootis set (the pre-boot drop the sample header currently asks the
reader to run by hand); the flusher and guard run as a controller-managed DaemonSet or a per-member
privileged sidecar, and the suspend is performed by the controller itself (it already holds the RBAC),
which removes the sample's ServiceAccount/Role.- Metrics in the existing Prometheus registry:
llmkube_multinode_member_memfree_bytes,
llmkube_multinode_page_cache_drops_total,llmkube_multinode_guard_suspends_total, so the
Grafana ring dashboard can show when and how often the guard acts. - Tests in
internal/controller/multinode_test.gofor the pod construction;make check-helm-rbaconly if
the controller's RBAC changes.
Alternatives Considered
- Keep the DaemonSet sample as is. It works (it is what serves the ring today) but it needs its own
RBAC, has no metrics, and is easy to forget on a new ring. - Ship the DaemonSet as an optional Helm chart template instead of a CRD field. Fewer moving parts
than a controller feature, but it still cannot sequence the pre-boot drop against the member pod.
Additional Context
- Related issues: #1856 (the recreate loop that hides a failed boot; a pinned boot on a dirty cache
would fail into it) - Similar features in other projects: tonyd2wild's
memfree_flusher.sh/servguard.shand
bot-lab-21'sdsv41_node_prep.sh(unconditional load-time flusher, MemFree gate) in the DGX Spark
DeepSeek-V4.1 recipes; both are host scripts. - Workarounds you're currently using:
config/samples/gb10-memory-guard.yaml(DaemonSet + scoped Role)
andconfig/samples/jobs/drop-page-cache.yaml.
Priority
- High - Would significantly improve my workflow
Willingness to Contribute
- Yes, I can submit a PR
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day