[Feature] Support `return_hidden_states` in async engine

(1 commentaire) (0 réaction) (0 personne assignée)Python (6 216 forks)auto 404

good first issue

Métriques du dépôt

Stars: (28 442 étoiles)
Métriques de merge PR: (Merge moyen 1j 13h) (1 000 PRs mergées en 30 j)

Description

Checklist

1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
2. Please use English, otherwise it will be closed.

Motivation

Currently, it is possible to get hidden states using the offline interface, i.e. Engine.generate, but Engine.async_generate doesn't support this function.

result = await model.async_generate(
        ["hello", "how are you doing?"],
        sampling_params={"temperature": 0, "max_new_tokens": 1},
        return_hidden_states=True)

TypeError: Engine.async_generate() got an unexpected keyword argument 'return_hidden_states'

This can be useful when developing web services where some additional hidden states postprocess logics is done.

Related resources

No response

Guide contributeur

Direction de recherche: Examinez comment return hidden states est traité dans Engine.generate et ajoutez le même support à async generate.
Stack technique: python
Domaine: backend
Type d'issue: Fonctionnalité
Difficulté: 2
Temps estimé: 1-3 heures
Statut d'activité: Active
Clarté: Claire
Prérequis: Python
Accessibilité débutant: 70