sgl-project/sglang

[Feature] Reduce constrained-decoding overhead in TP

Open

#13,809 opened on Nov 23, 2025

View on GitHub
 (1 comment) (0 reactions) (1 assignee)Python (28,442 stars) (6,216 forks)auto 404
good first issue

Description

Checklist

Motivation

Currently, when tensor-parallelism (TP) and constrained-decoding are enabled, each TP worker will compile the same grammar across different TP ranks. This will incur non-trivial CPU overhead (for TP $n$, the overhead is $n \times$. In fact, we only need to compile the grammar and sample the result on the first rank.

A possible implementation can be:

  1. Only apply grammar token mask on the rank 0.
  2. Broadcast the next tokens id from rank 0 when there're grammars in batch.
  3. Keep the old code path when there's no grammar in batch (i.e. no extra all-reduce).

https://github.com/sgl-project/sglang/blob/5c2915494c83f076a11afb2c3382eeb8a41f1974/python/sglang/srt/layers/sampler.py#L205-L217

Related resources

No response

Contributor guide

Tech stack
python
Domain
backend
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Implement the proposed approach: compile grammar only on rank 0, broadcast next token ids when grammars are present, and keep current code path when no grammars. Refer to the sampler.py code linked.