Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Observability for LiveKit Inference TTS fallback (which model served the request), especially in expressive mode

オープン
  • 何か問題がありますか？
#8,064 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
38/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
backend-api-design, observability

調査の方向性

Start by reading inference/tts.py and tracing how it handles the session.created payload and populates tts_node spans, TTSModelUsage, and TTSMetrics; compare this with _record_fallback_served in tts.FallbackAdapter. The issue is done when per-request observability identifies whether a fallback served audio and which model served it, with corresponding usage attribution; the payload also identifies analogous STT fallback behavior and expressive-mode markup as related concerns.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Feature Type

Would make my life easier

Feature Description

We use expressive mode with an inference.TTS primary and server-side fallbacks:

tts = inference.TTS(
    model="xai/tts-1",
    voice="ursa",
    language="ja",
    fallback=[
        {"model": "inworld/inworld-tts-2", "voice": "Asuka"},
        {"model": "cartesia/sonic-3.6", "voice": "<voice id>"},
        {"model": "google/gemini-3.8-flash-tts", "voice": "Achernar"},
    ],
)
session = AgentSession(tts=tts, expressive=True, ...)

Server-side fallback is the only option that keeps expressive mode on. Expressive mode requires the session TTS to be an inference.TTS, so wrapping providers in tts.FallbackAdapter disables it.

It is currently unclear to me how we would know that a failure and fallback has happened at all. As far as I can tell, nothing on the agent side reveals it, As far as I am aware:

  • The gateway only sends session.created, output_audio, output_alignment, done and error. None of these says which model produced the audio. error only arrives when every model has failed.
  • No AgentSession error event fires when Inference falls back successfully. The Fallback strategies page says "When a fallback is triggered, AgentSession emits an error event", but that does not seem to apply to the Inference Fallback Adapter.
  • The tts_node span's gen_ai.request.model, TTSModelUsage.model and metrics all report the primary model, because they come from inference.TTS._opts.model.

We reproduced this by giving xai/tts-1 a voice xAI rejects. With no fallback configured the request fails (xAI TTS error: ... Voice '...' not found). With an Inworld fallback configured, the session plays audio normally, logs zero TTS errors, and the trace and usage still say xai/tts-1.

We'd like to be able to tell, per request, that a fallback happened and which model served it. That would let us alert on provider outages, attribute cost correctly, and debug voice or quality differences users report. Possible shapes:

  1. The gateway reports the serving model and voice (for example in session.created or done, or as a new fallback message with the failed model and error). inference.TTS then exposes it.
  2. The tts_node / tts_request spans get gen_ai.response.model, lk.fallback.label and lk.fallback.index, matching what tts.FallbackAdapter already records via _record_fallback_served.
  3. Usage and metrics (TTSModelUsage, TTSMetrics) are attributed to the model that served, so per-provider usage is correct.
  4. An AgentSession event (or a non-fatal error event with recoverable=True) fires when Inference falls back, as the docs describe.

The same would be useful for inference.STT fallbacks.

Related Concern

  • Related expressive-mode concern: is ti the case that markup is converted to the primary model's dialect on the client (markup.convert, keyed on _provider_key() of the primary model) before Inference falls back? A fallback provider therefore receives the primary's native tags, for example [pause] and <soft>…</soft> from xAI sent to Inworld. Without knowing a fallback happened, we can't detect or correct for this either. If Inference reported the serving model, or converted the markup server-side for the provider that actually serves, both problems would be solved.
Workarounds / Alternatives
  • tts.FallbackAdapter gives this visibility, but it disables expressive mode, so it isn't an option for us.
  • Checking per-model usage in LiveKit Cloud after the fact, if the dashboard breaks usage down by provider. This doesn't work per session and can't be used for alerting.
Additional Context
  • livekit-agents 1.8.2 (Python). The code paths above look the same on current main (inference/tts.py still ignores the session.created payload).
主要言語
Python
スター
14.6k
フォーク
3.9k
平均マージ
3日 21時間
マージ済み PR（30日）
197

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

livekit/agents のほかの issue

livekit/agents の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。