[FEATURE] multiNode: operator-owned page-cache discipline and MemFree guard for unified-memory (GB10) members

Open
#1,857 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes

Research direction

Start with api/v1alpha1/inferenceservice_types.go and internal/controller/multinode.go to trace MultiNodeSpec and member pod construction. Read internal/controller/multinode_test.go and inspect config/samples/gb10-memory-guard.yaml for existing behavior. Done means the operator can configure pre-boot cache drops, serving memory guards, and sysctls for each member, with the proposed metrics and tests.

Written by the indexing model from the issue text.

Description

enhancement

Feature Description

When a multiNode vLLM group pins its KV cache with --kv-cache-memory-bytes, vLLM skips memory
profiling entirely (its log: "reserved N GiB memory for KV Cache as specified by kv_cache_memory_bytes
config and skipped memory profiling. This does not respect the gpu_memory_utilization config"). On a
DGX Spark host RAM is GPU memory, so nothing then protects the group from a dirty page cache at boot
or from the page cache regrowing while serving (Engram row reads on DeepSeek-V4.1 do exactly that).
Today that protection is a standalone privileged DaemonSet sample with its own ServiceAccount and Role
(config/samples/gb10-memory-guard.yaml). The operator should own it.

Problem Statement

As a maintainer running DeepSeek-V4.1-Flash EXL3 on a three-Spark ring at 512K context with vision,
I want the operator to guarantee the pre-boot cache drop and the serving-phase memory guard for a
pinned multiNode group, so a boot cannot start on a dirty cache and a member cannot starve the GPU
allocator, without applying a second manifest with separate RBAC.

Measured on 2026-09-20 (three GB10 members, 512K, vision, KV pin 4.2 GiB per rank):

condition during a 450,977-token prefill lowest MemFree on a rank
no flusher 1.0 GiB
a 15 s flusher loop 2.7 GiB
the guard DaemonSet (2 s loop) plus vm.min_free_kbytes=1048576, vm.watermark_scale_factor=200 no drop needed, ~10 GiB after

The nodes shipped at vm.min_free_kbytes 44 MiB and watermark_scale_factor 10. A fresh 460 GB
checkpoint stage also leaves ~10 GiB of page cache resident even with the group suspended.

Proposed Solution

A memory block on spec.multiNode that the controller implements for every member:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
spec:
  multiNode:
    memory:
      dropPageCacheBeforeBoot: true          # privileged step on the member's node right before its pod starts
      flushBelowMemFreeGiB: 4                # serving-phase flusher (drop caches when MemFree falls under this)
      suspendBelowMemFreeGiB: 1.5            # guard: suspend the group once if MemFree stays under this
      sysctls:                                # applied on the member node, logged before/after
        vm.min_free_kbytes: "1048576"
        vm.watermark_scale_factor: "200"

Implementation sketch:

  • api/v1alpha1/inferenceservice_types.go: MultiNodeSpec (line 271) gains Memory *MultiNodeMemory;
    make manifests then make chart-crds.
  • internal/controller/multinode.go: constructMemberPods adds a privileged init container that drops
    caches when dropPageCacheBeforeBoot is set (the pre-boot drop the sample header currently asks the
    reader to run by hand); the flusher and guard run as a controller-managed DaemonSet or a per-member
    privileged sidecar, and the suspend is performed by the controller itself (it already holds the RBAC),
    which removes the sample's ServiceAccount/Role.
  • Metrics in the existing Prometheus registry: llmkube_multinode_member_memfree_bytes,
    llmkube_multinode_page_cache_drops_total, llmkube_multinode_guard_suspends_total, so the
    Grafana ring dashboard can show when and how often the guard acts.
  • Tests in internal/controller/multinode_test.go for the pod construction; make check-helm-rbac only if
    the controller's RBAC changes.

Alternatives Considered

  • Keep the DaemonSet sample as is. It works (it is what serves the ring today) but it needs its own
    RBAC, has no metrics, and is easy to forget on a new ring.
  • Ship the DaemonSet as an optional Helm chart template instead of a CRD field. Fewer moving parts
    than a controller feature, but it still cannot sequence the pre-boot drop against the member pod.

Additional Context

  • Related issues: #1856 (the recreate loop that hides a failed boot; a pinned boot on a dirty cache
    would fail into it)
  • Similar features in other projects: tonyd2wild's memfree_flusher.sh / servguard.sh and
    bot-lab-21's dsv41_node_prep.sh (unconditional load-time flusher, MemFree gate) in the DGX Spark
    DeepSeek-V4.1 recipes; both are host scripts.
  • Workarounds you're currently using: config/samples/gb10-memory-guard.yaml (DaemonSet + scoped Role)
    and config/samples/jobs/drop-page-cache.yaml.

Priority

  • High - Would significantly improve my workflow

Willingness to Contribute

  • Yes, I can submit a PR
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.