sgl-project/sglang

[Feature] Support `return_hidden_states` in async engine

已关闭

#6,528 创建于 2025年5月22日

 (1 条评论) (0 个反应) (0 位负责人)Python (6,216 个派生)auto 404
good first issue

仓库指标

星标
 (28,442 个星标)
PR 合并指标
 (平均合并 2天 1小时) (30 天内合并 1,000 个 PR)

描述

Checklist

Motivation

Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.

result = await model.async_generate(
        ["hello", "how are you doing?"],
        sampling_params={"temperature": 0, "max_new_tokens": 1},
        return_hidden_states=True)
TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'

This can be useful when developing web services where some additional hidden states postprocess logics is done.

Related resources

No response

贡献者指南