[Feature] Support `return_hidden_states` in async engine

(1 commento) (0 reazioni) (0 assegnatari)Python (6216 fork)auto 404

good first issue

Metriche repository

Star: (28.442 stelle)
Metriche merge PR: (Merge medio 2g 1h) (1000 PR mergiate in 30 g)

Descrizione

Checklist

1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
2. Please use English, otherwise it will be closed.

Motivation

Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.

result = await model.async_generate(
        ["hello", "how are you doing?"],
        sampling_params={"temperature": 0, "max_new_tokens": 1},
        return_hidden_states=True)

TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'

This can be useful when developing web services where some additional hidden states postprocess logics is done.

Related resources

No response

Guida contributor

Direzione di ricerca: Esamina come viene gestito return hidden states in Engine.generate e aggiungi lo stesso supporto a async generate.
Tech stack: python
Dominio: backend
Tipo issue: Funzionalità
Difficoltà: 2
Tempo stimato: 1-3 ore
Stato attività: Attiva
Chiarezza: Chiara
Prerequisiti: Python
Adatta ai principianti: 70