good first issue
仓库指标
- 星标
- (28,442 个星标)
- PR 合并指标
- (平均合并 2天 1小时) (30 天内合并 1,000 个 PR)
描述
Checklist
- 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 2. Please use English, otherwise it will be closed.
Motivation
Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.
result = await model.async_generate(
["hello", "how are you doing?"],
sampling_params={"temperature": 0, "max_new_tokens": 1},
return_hidden_states=True)
TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'
This can be useful when developing web services where some additional hidden states postprocess logics is done.
Related resources
No response