Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Observability for LiveKit Inference TTS fallback (which model served the request), especially in expressive mode

Abierto
  • ¿Algo no está bien?
#8,064 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend-api-design, observability

Línea de trabajo

Start by reading inference/tts.py and tracing how it handles the session.created payload and populates tts_node spans, TTSModelUsage, and TTSMetrics; compare this with _record_fallback_served in tts.FallbackAdapter. The issue is done when per-request observability identifies whether a fallback served audio and which model served it, with corresponding usage attribution; the payload also identifies analogous STT fallback behavior and expressive-mode markup as related concerns.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
Feature Type

Would make my life easier

Feature Description

We use expressive mode with an inference.TTS primary and server-side fallbacks:

tts = inference.TTS(
    model="xai/tts-1",
    voice="ursa",
    language="ja",
    fallback=[
        {"model": "inworld/inworld-tts-2", "voice": "Asuka"},
        {"model": "cartesia/sonic-3.6", "voice": "<voice id>"},
        {"model": "google/gemini-3.8-flash-tts", "voice": "Achernar"},
    ],
)
session = AgentSession(tts=tts, expressive=True, ...)

Server-side fallback is the only option that keeps expressive mode on. Expressive mode requires the session TTS to be an inference.TTS, so wrapping providers in tts.FallbackAdapter disables it.

It is currently unclear to me how we would know that a failure and fallback has happened at all. As far as I can tell, nothing on the agent side reveals it, As far as I am aware:

  • The gateway only sends session.created, output_audio, output_alignment, done and error. None of these says which model produced the audio. error only arrives when every model has failed.
  • No AgentSession error event fires when Inference falls back successfully. The Fallback strategies page says "When a fallback is triggered, AgentSession emits an error event", but that does not seem to apply to the Inference Fallback Adapter.
  • The tts_node span's gen_ai.request.model, TTSModelUsage.model and metrics all report the primary model, because they come from inference.TTS._opts.model.

We reproduced this by giving xai/tts-1 a voice xAI rejects. With no fallback configured the request fails (xAI TTS error: ... Voice '...' not found). With an Inworld fallback configured, the session plays audio normally, logs zero TTS errors, and the trace and usage still say xai/tts-1.

We'd like to be able to tell, per request, that a fallback happened and which model served it. That would let us alert on provider outages, attribute cost correctly, and debug voice or quality differences users report. Possible shapes:

  1. The gateway reports the serving model and voice (for example in session.created or done, or as a new fallback message with the failed model and error). inference.TTS then exposes it.
  2. The tts_node / tts_request spans get gen_ai.response.model, lk.fallback.label and lk.fallback.index, matching what tts.FallbackAdapter already records via _record_fallback_served.
  3. Usage and metrics (TTSModelUsage, TTSMetrics) are attributed to the model that served, so per-provider usage is correct.
  4. An AgentSession event (or a non-fatal error event with recoverable=True) fires when Inference falls back, as the docs describe.

The same would be useful for inference.STT fallbacks.

Related Concern

  • Related expressive-mode concern: is ti the case that markup is converted to the primary model's dialect on the client (markup.convert, keyed on _provider_key() of the primary model) before Inference falls back? A fallback provider therefore receives the primary's native tags, for example [pause] and <soft>…</soft> from xAI sent to Inworld. Without knowing a fallback happened, we can't detect or correct for this either. If Inference reported the serving model, or converted the markup server-side for the provider that actually serves, both problems would be solved.
Workarounds / Alternatives
  • tts.FallbackAdapter gives this visibility, but it disables expressive mode, so it isn't an option for us.
  • Checking per-model usage in LiveKit Cloud after the fact, if the dashboard breaks usage down by provider. This doesn't work per session and can't be used for alerting.
Additional Context
  • livekit-agents 1.8.2 (Python). The code paths above look the same on current main (inference/tts.py still ignores the session.created payload).
Lenguaje dominante
Python
Estrellas
14.6k
Forks
3.9k
Merge medio
3 d 21 h
PR fusionados (30 d)
197

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de livekit/agents

Todos los issues de livekit/agents

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.