sgl-project/sglang

[Feature] Support piggyback server load report.

開放

#11,186 建立於 2025年10月3日

 (4 則留言) (0 個反應) (1 位負責人)Python (6,216 個分叉)auto 404
good first issue

倉庫指標

星標
 (28,442 顆星)
PR 合併指標
 (平均合併 1天 13小時) (30 天內合併 1,000 個 PR)

描述

Checklist

Motivation

To support a flexible and reliable load balancing strategy, external router (sgl-router) and internal controller (dp-controller) should be aware of the real-time load situation.

Current implementation uses a polling mechanism and a watch-based communicator to query load from the scheduler. However, this is flexible and elegant.

  • Implement a piggyback load reporting mechanism, in which load information is carried through restreaming token messages.
  • Also keep the watching mode, always storing the latest load information in http worker for external query.

Related resources

No response

貢獻者指南