docs: add multi-node inference without Ray (mp backend) guide
Maintainers usually reply within 4 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 76/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Quiet
Research direction
Start with docs/model-serving/generative-inference/multi-node/multi-node.md and the related PR #5366 to understand the existing Ray-based guide. Document the mp annotation, PP/TP sizing, worker headless service, injected environment variables, and add a complete multi-node InferenceService YAML example; done means the guide covers KServe v0.18 mp deployments without Ray.
Written by the indexing model from the issue text.
Description
Summary
KServe v0.18 added support for multi-node InferenceService deployments without requiring Ray, using vLLM's mp (multiprocessing) distributed executor backend. The current multi-node documentation at docs/model-serving/generative-inference/multi-node/multi-node.md only covers Ray-based multi-node inference.
What needs to be documented
- The
multinode/executor-backendannotation on ServingRuntime spec (rayvsmpmode) - How
mpmode derives the number of nodes from pipeline parallelism (PP) and GPUs per node from tensor parallelism (TP) - The worker headless service for pod DNS discovery
- Environment variables injected into worker containers (
PIPELINE_PARALLEL_SIZE,TENSOR_PARALLEL_SIZE,WORKER_SVC) - Example YAML for a multi-node InferenceService using
mpbackend
Related PR
Since
KServe v0.18
- Dominant language
- MDX
- Stars
- 113
- Forks
- 200
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 9
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from kserve/website
-
Difficulty 2/5 1-2 days Newbie friendliness 72/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
kserve/website#725 · 1 comment ·
Maintainers usually reply within 4 days
Similar issues
-
fix(cluster): keep a panicking encode of a remote reply from failing the responderPossibly taken @cosmir17 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
tellus-rs/tellus#312 · 2 comments ·
Maintainers usually reply within 1 day
-
area/build area/events area/storage area/storage/kafkasql area/storage/sql
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Apicurio/apicurio-registry#11079 ·
Maintainers usually reply within 1 day
-
bug: PriorityRouter always picks the first engine when several engines have the same lowest loadPossibly taken @Scanf-s claimed this 1 day ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
vllm-project/production-stack#1203 ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
celery/kombu#2840 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
volcano-sh/volcano#6292 · 1 comment ·
Maintainers usually reply within 1 day