[FEATURE] multiNode: llama.cpp RPC members (rank 0 --rpc, workers ggml-rpc-server)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
Research direction
Read internal/controller/runtime.go and runtime_vllm.go to understand the existing MultiNodeArgsBuilder pattern, then inspect runtime_llamacpp.go for the CommandBuilder seam. Trace validation and GPU sharing through internal/controller/multinode.go, internal/controller/gpu_sharing.go, and api/v1alpha1/inferenceservice_types.go; review the two-node RPC guide and existing samples. Done when llama.cpp rank 0 and worker pods are generated and validated for multi-node inference, with the manual guide retired and a second sample added.
Written by the indexing model from the issue text.
Description
Feature Description
Add llama.cpp as a spec.multiNode backend: rank 0 runs llama-server --rpc <workers>, ranks 1..N run ggml-rpc-server, using the group machinery #1746 shipped for vLLM.
Problem Statement
MultiNodeArgsBuilder (internal/controller/runtime.go) is implemented only by VLLMBackend (runtime_vllm.go), and validateMultiNode (internal/controller/multinode.go) rejects every other runtime. The two-Spark llama.cpp RPC pattern in docs/dgx-spark-two-node-rpc.md still means two hand-written pods, and it is the only path for models vLLM does not serve on this hardware (GGUF-only checkpoints, TurboQuant KV, the cuda-gb10 image's RPC build).
Proposed Solution
LlamaCppBackendimplementsMultiNodeArgsBuilder: rank 0 gets--rpc <addr1>:<port>,...built from every other member'sfabric.addressand arendezvousPort-derived RPC port; ranks 1..N replace the server command withggml-rpc-server --host <fabric.address> --port <port>(aCommandBuilderseam exists already; the worker needs no model args and no probes, whichmemberPodFromTemplatealready strips).- Readiness: the head's existing probes; workers Running is enough, as with vLLM.
- Guard: the vLLM world-size check does not apply; llama.cpp splits layers by
--tensor-splitor the RPC device order, so the guard skipstensorParallelSizefor this runtime and the CEL rule onInferenceServiceSpecwidens toruntime in ['vllm','llamacpp']. - The memory note from the study applies: RPC workers do not free memory on disconnect, so a recreate must also recreate the worker pods, which the group already does.
Files: internal/controller/runtime_llamacpp.go (or wherever LlamaCppBackend lives), internal/controller/multinode.go (validateMultiNode), internal/controller/gpu_sharing.go (guard), api/v1alpha1/inferenceservice_types.go (CEL message), docs/dgx-spark-two-node-rpc.md (retire the manual pattern), a second sample.
Additional Context
Parent: #1423. Shipped v1: #1746. Measured on the ring: llama.cpp RPC 357 tok/s prefill and 19.4 decode against vLLM's 2,089 and 24.0 for the same model family, so this is about coverage, not speed.
Priority
- Low - Would be nice
Willingness to Contribute
- Yes, I can submit a PR
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day