sgl-project/sglang
[Feature] Support piggyback server load report.
Offen
#11.186 geöffnet am 03.10.2025
good first issue
Repository-Metriken
- Stars
- (28.442 Sterne)
- PR-Merge-Metriken
- (Durchschn. Merge 1T 13h) (1.000 gemergte PRs in 30 T)
Beschreibung
Checklist
- 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 2. Please use English, otherwise it will be closed.
Motivation
To support a flexible and reliable load balancing strategy, external router (sgl-router) and internal controller (dp-controller) should be aware of the real-time load situation.
Current implementation uses a polling mechanism and a watch-based communicator to query load from the scheduler. However, this is flexible and elegant.
- Implement a piggyback load reporting mechanism, in which load information is carried through restreaming token messages.
- Also keep the watching mode, always storing the latest load information in http worker for external query.
Related resources
No response