sgl-project/sglang

[Feature] Support `return_hidden_states` in async engine

クローズ

#6,528 opened on 2025/05/22

 (1 件のコメント) (0 件のリアクション) (0 人の担当者)Python (6,216 件のフォーク)auto 404
good first issue

Repository metrics

Stars
 (28,442 個のスター)
PR merge metrics
 (平均マージ 2d 1h) (30d で 1,000 merged PRs)

説明

Checklist

Motivation

Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.

result = await model.async_generate(
        ["hello", "how are you doing?"],
        sampling_params={"temperature": 0, "max_new_tokens": 1},
        return_hidden_states=True)
TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'

This can be useful when developing web services where some additional hidden states postprocess logics is done.

Related resources

No response

コントリビューターガイド