[FEATURE] multiNode: llama.cpp RPC members (rank 0 --rpc, workers ggml-rpc-server)

Open
#1,753 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
go, kubernetes

Research direction

Read internal/controller/runtime.go and runtime_vllm.go to understand the existing MultiNodeArgsBuilder pattern, then inspect runtime_llamacpp.go for the CommandBuilder seam. Trace validation and GPU sharing through internal/controller/multinode.go, internal/controller/gpu_sharing.go, and api/v1alpha1/inferenceservice_types.go; review the two-node RPC guide and existing samples. Done when llama.cpp rank 0 and worker pods are generated and validated for multi-node inference, with the manual guide retired and a second sample added.

Written by the indexing model from the issue text.

Description

enhancement

Feature Description

Add llama.cpp as a spec.multiNode backend: rank 0 runs llama-server --rpc <workers>, ranks 1..N run ggml-rpc-server, using the group machinery #1746 shipped for vLLM.

Problem Statement

MultiNodeArgsBuilder (internal/controller/runtime.go) is implemented only by VLLMBackend (runtime_vllm.go), and validateMultiNode (internal/controller/multinode.go) rejects every other runtime. The two-Spark llama.cpp RPC pattern in docs/dgx-spark-two-node-rpc.md still means two hand-written pods, and it is the only path for models vLLM does not serve on this hardware (GGUF-only checkpoints, TurboQuant KV, the cuda-gb10 image's RPC build).

Proposed Solution

  • LlamaCppBackend implements MultiNodeArgsBuilder: rank 0 gets --rpc <addr1>:<port>,... built from every other member's fabric.address and a rendezvousPort-derived RPC port; ranks 1..N replace the server command with ggml-rpc-server --host <fabric.address> --port <port> (a CommandBuilder seam exists already; the worker needs no model args and no probes, which memberPodFromTemplate already strips).
  • Readiness: the head's existing probes; workers Running is enough, as with vLLM.
  • Guard: the vLLM world-size check does not apply; llama.cpp splits layers by --tensor-split or the RPC device order, so the guard skips tensorParallelSize for this runtime and the CEL rule on InferenceServiceSpec widens to runtime in ['vllm','llamacpp'].
  • The memory note from the study applies: RPC workers do not free memory on disconnect, so a recreate must also recreate the worker pods, which the group already does.

Files: internal/controller/runtime_llamacpp.go (or wherever LlamaCppBackend lives), internal/controller/multinode.go (validateMultiNode), internal/controller/gpu_sharing.go (guard), api/v1alpha1/inferenceservice_types.go (CEL message), docs/dgx-spark-two-node-rpc.md (retire the manual pattern), a second sample.

Additional Context

Parent: #1423. Shipped v1: #1746. Measured on the ring: llama.cpp RPC 357 tok/s prefill and 19.4 decode against vLLM's 2,089 and 24.0 for the same model family, so this is about coverage, not speed.

Priority

  • Low - Would be nice

Willingness to Contribute

  • Yes, I can submit a PR
Dominant language
Go
Stars
232
Forks
37
Avg merge
21h 42m
Merged PRs (30d)
244

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from defilantech/LLMKube

All issues in defilantech/LLMKube

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.