Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Observability for LiveKit Inference TTS fallback (which model served the request), especially in expressive mode

Open
  • Is something wrong?
#8,064 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend-api-design, observability

Research direction

Start by reading inference/tts.py and tracing how it handles the session.created payload and populates tts_node spans, TTSModelUsage, and TTSMetrics; compare this with _record_fallback_served in tts.FallbackAdapter. The issue is done when per-request observability identifies whether a fallback served audio and which model served it, with corresponding usage attribution; the payload also identifies analogous STT fallback behavior and expressive-mode markup as related concerns.

Written by the indexing model from the issue text.

Description

enhancement
Feature Type

Would make my life easier

Feature Description

We use expressive mode with an inference.TTS primary and server-side fallbacks:

tts = inference.TTS(
    model="xai/tts-1",
    voice="ursa",
    language="ja",
    fallback=[
        {"model": "inworld/inworld-tts-2", "voice": "Asuka"},
        {"model": "cartesia/sonic-3.6", "voice": "<voice id>"},
        {"model": "google/gemini-3.8-flash-tts", "voice": "Achernar"},
    ],
)
session = AgentSession(tts=tts, expressive=True, ...)

Server-side fallback is the only option that keeps expressive mode on. Expressive mode requires the session TTS to be an inference.TTS, so wrapping providers in tts.FallbackAdapter disables it.

It is currently unclear to me how we would know that a failure and fallback has happened at all. As far as I can tell, nothing on the agent side reveals it, As far as I am aware:

  • The gateway only sends session.created, output_audio, output_alignment, done and error. None of these says which model produced the audio. error only arrives when every model has failed.
  • No AgentSession error event fires when Inference falls back successfully. The Fallback strategies page says "When a fallback is triggered, AgentSession emits an error event", but that does not seem to apply to the Inference Fallback Adapter.
  • The tts_node span's gen_ai.request.model, TTSModelUsage.model and metrics all report the primary model, because they come from inference.TTS._opts.model.

We reproduced this by giving xai/tts-1 a voice xAI rejects. With no fallback configured the request fails (xAI TTS error: ... Voice '...' not found). With an Inworld fallback configured, the session plays audio normally, logs zero TTS errors, and the trace and usage still say xai/tts-1.

We'd like to be able to tell, per request, that a fallback happened and which model served it. That would let us alert on provider outages, attribute cost correctly, and debug voice or quality differences users report. Possible shapes:

  1. The gateway reports the serving model and voice (for example in session.created or done, or as a new fallback message with the failed model and error). inference.TTS then exposes it.
  2. The tts_node / tts_request spans get gen_ai.response.model, lk.fallback.label and lk.fallback.index, matching what tts.FallbackAdapter already records via _record_fallback_served.
  3. Usage and metrics (TTSModelUsage, TTSMetrics) are attributed to the model that served, so per-provider usage is correct.
  4. An AgentSession event (or a non-fatal error event with recoverable=True) fires when Inference falls back, as the docs describe.

The same would be useful for inference.STT fallbacks.

Related Concern

  • Related expressive-mode concern: is ti the case that markup is converted to the primary model's dialect on the client (markup.convert, keyed on _provider_key() of the primary model) before Inference falls back? A fallback provider therefore receives the primary's native tags, for example [pause] and <soft>…</soft> from xAI sent to Inworld. Without knowing a fallback happened, we can't detect or correct for this either. If Inference reported the serving model, or converted the markup server-side for the provider that actually serves, both problems would be solved.
Workarounds / Alternatives
  • tts.FallbackAdapter gives this visibility, but it disables expressive mode, so it isn't an option for us.
  • Checking per-model usage in LiveKit Cloud after the fact, if the dashboard breaks usage down by provider. This doesn't work per session and can't be used for alerting.
Additional Context
  • livekit-agents 1.8.2 (Python). The code paths above look the same on current main (inference/tts.py still ignores the session.created payload).
Dominant language
Python
Stars
14.6k
Forks
3.9k
Avg merge
3d 19h
Merged PRs (30d)
204

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from livekit/agents

All issues in livekit/agents

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.