sgl-project/sglang
[Feature] Reduce constrained-decoding overhead in TP
Aberta
#13.809 aberto em 23 de nov. de 2025
good first issue
Métricas do repositório
- Stars
- (28.442 estrelas)
- Métricas de merge de PR
- (Mesclagem média 1d 13h) (1.000 fundiu PRs em 30d)
Description
Checklist
- If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- Please use English. Otherwise, it will be closed.
Motivation
Currently, when tensor-parallelism (TP) and constrained-decoding are enabled, each TP worker will compile the same grammar across different TP ranks. This will incur non-trivial CPU overhead (for TP $n$, the overhead is $n \times$. In fact, we only need to compile the grammar and sample the result on the first rank.
A possible implementation can be:
- Only apply grammar token mask on the rank 0.
- Broadcast the next tokens id from rank 0 when there're grammars in batch.
- Keep the old code path when there's no grammar in batch (i.e. no extra all-reduce).
Related resources
No response