sgl-project/sglang

[Feature] Enable EPLB in Draft Models

Open

#7,893 opened on Jul 9, 2025

View on GitHub
 (6 comments) (0 reactions) (1 assignee)Python (6,216 forks)auto 404
good first issue

Repository metrics

Stars
 (28,442 stars)
PR merge metrics
 (Avg merge 2d 1h) (1,000 merged PRs in 30d)

Description

Checklist

Motivation

Currently, EPLB is not supported in draft models, which constrains the parallelism size. For example, with EPLB, we can set the EP size as 72/144 for the DeepSeek model as it has 288 experts in total. However, these parallelism settings cannot be adopted when MTP is enabled as the draft model as has 256 experts.

Related resources

No response

Contributor guide

Research direction
Study the existing EPLB implementation for non draft models in the sglang codebase. Identify the key components (e.g., expert routing, load balancing logic) and understand how they interact with the model architecture. Then, adapt these components to support draft models, considering differences such as the number of experts and model structure. Test the changes with appropriate configurations (e.g., DeepSeek with MTP).
Tech stack
python
Domain
backendinfrastructuremachine learning
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonDistributed computingExpert parallelismLLM serving
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40