[Feature] Support `return_hidden_states` in async engine

(1 Kommentar) (0 Reaktionen) (0 zugewiesene Personen)Python (6.216 Forks)auto 404

good first issue

Repository-Metriken

Stars: (28.442 Sterne)
PR-Merge-Metriken: (Durchschn. Merge 1T 13h) (1.000 gemergte PRs in 30 T)

Beschreibung

Checklist

1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
2. Please use English, otherwise it will be closed.

Motivation

Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.

result = await model.async_generate(
        ["hello", "how are you doing?"],
        sampling_params={"temperature": 0, "max_new_tokens": 1},
        return_hidden_states=True)

TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'

This can be useful when developing web services where some additional hidden states postprocess logics is done.

Related resources

No response

Contributor Guide

Research-Richtung: Untersuche, wie return hidden states in Engine.generate behandelt wird, und füge die gleiche Unterstützung zu async generate hinzu.
Tech Stack: python
Domain: backend
Issue Type: Funktion
Schwierigkeit: 2
Geschätzte Zeit: 1-3 Stunden
Aktivitätsstatus: Aktiv
Klarheit: Klar
Voraussetzungen: Python
Einsteigerfreundlichkeit: 70