sgl-project/sglang
[Feature] Reduce constrained-decoding overhead in TP
Ouverte
#13 809 ouverte le 23 nov. 2025
good first issue
Métriques du dépôt
- Stars
- (28 442 étoiles)
- Métriques de merge PR
- (Merge moyen 1j 13h) (1 000 PRs mergées en 30 j)
Description
Checklist
- If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Motivation
Currently, when tensor-parallelism (TP) and constrained-decoding are enabled, each TP worker will compile the same grammar across different TP ranks. This will incur non-trivial CPU overhead (for TP $n$, the overhead is $n \times$. In fact, we only need to compile the grammar and sample the result on the first rank.
A possible implementation can be:
- Only apply grammar token mask on the rank 0.
- Broadcast the next tokens id from rank 0 when there're grammars in batch.
- Keep the old code path when there's no grammar in batch (i.e. no extra all-reduce).
Related resources
No response