Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

feat(hf): serve concurrent sessions with batched generation

Abierto
  • ¿Algo no está bien?
#7,764 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
huggingface, python, pytorch
Área
ai, backend

Línea de trabajo

Start with livekit/plugins/hf/llm.py and inspect _LoadedModel.generate_lock and the existing generation path. Compare the supported batching options in the installed transformers version with the proposed in-plugin scheduler. Done means concurrent sessions share generation work while LLMStream streaming, cancellation, and tool call behavior stay unchanged; the issue does not identify tests or a specific batching design.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Feature Type

Would make my life easier

Feature Description

livekit-plugins-hf (#7731) loads one copy of the model per process and shares it across sessions. Generation is serialized by a per-model lock, so sessions on the same worker take turns: each caller waits for every reply ahead of it to finish.

Measured on Qwen2.5-0.5B-Instruct, Apple M4 (MPS), temperature=0, 3 sessions sending the same request at the same time:

time to first token total
one session alone 99 ms 481 ms
session 1 of 3 166 ms 503 ms
session 2 of 3 603 ms 1343 ms
session 3 of 3 1568 ms 2448 ms

Time to first token grows with every session queued ahead. In a voice agent, that is dead air on the line.

Proposal: batch concurrent requests into shared forward passes, so N sessions decode together instead of in sequence. Decoding is memory-bandwidth bound, so a small batch costs little more per step than a single sequence. Options:

  • transformers continuous batching (generate_batch / the paged-attention cache) where the installed version supports it
  • a small in-plugin scheduler that pads and steps active requests together

The streaming, cancellation and tool call behaviour of LLMStream stays unchanged.

Workarounds / Alternatives
  • One model per job process (the default process executor). This multiplies memory by the number of jobs, so a 1.5B to 7B model fits only a few sessions per GPU.
  • Run vLLM or TGI as a separate server and use the openai plugin with base_url. This works, but it adds a separate service, which the local plugin exists to avoid.
Additional Context
  • Follow-up to #7731 (the plugin) and part of #5351 (local HF LLM, where KV cache compression is also proposed to raise sessions per GPU).
  • The lock is in livekit/plugins/hf/llm.py: _LoadedModel.generate_lock.
Lenguaje dominante
Python
Estrellas
14.6k
Forks
3.9k
Merge medio
3 d 1 h
PR fusionados (30 d)
173

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de livekit/agents

Todos los issues de livekit/agents

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.