Expressive Mode: `speech_steering` always injects an English disfluency line, and that text cannot be customized

オープン
  • 何か問題がありますか？
#7,554 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@rrfunde がすでに取り組んでいます。

2026年10月14日 から。

  • #7824 @rrfunde による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
54/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
python
領域
ai-infra-agents

調査の方向性

Start with steering_instructions() and resolve_expressive_options in livekit/agents/tts/_provider_format.py, then trace how AgentActivity._inject_expressive_instructions adds the rendered text to chat context. Check the existing expressive-mode tests and run the relevant tests before changing behavior. Done means callers can customize steering prose and disfluency text while English defaults and vocabulary filtering remain unchanged.

索引モデルが issue の本文から書いたものです。

説明

enhancement
Feature Type

Would make my life easier

Feature Description

tts_instructions_template and tts_instructions_append can replace or extend the expressive markup guide. The delivery line that speech_steering injects cannot. It is hardcoded English, and resolve_expressive_options appends it after any custom template.

steering_instructions() in livekit/agents/tts/_provider_format.py renders disfluencies as one of two fixed sentences:

  • True: Sprinkle in natural fillers (um, uh) and openers (oh, well, so), zero to two per turn, never mechanical.
  • False: No fillers (um, uh). Sound composed and fluent.

DEFAULT_EXPRESSIVE_OPTIONS sets disfluencies=True. resolve_expressive_options merges that default, so the English line is injected whenever expressive mode is on, including expressive=True and a custom tts_instructions_template that never mentions fillers. Omitting {tts.markup.llm_instructions} skips the provider markup block. It does not skip this line. AgentActivity._inject_expressive_instructions then puts the rendered text in the LLM chat context.

That breaks non-English agents. A Japanese agent that wants えっと、あの、まあ、そうですね still sees um, uh and oh, well, so in the same prompt. Models follow the English examples. Setting disfluencies=False does not help if you still want fillers. It only swaps in the other English sentence.

Please make every expressive prompt fragment customizable, the same way the markup guide already is. The disfluency line is the one we need to edit. A language-appropriate filler list should be able to replace um, uh without rewriting LiveKit's tag vocabulary.

Two changes would cover it:

  1. Stop appending steering_instructions() unconditionally. Expose that fragment as a placeholder, the way {tts.markup.llm_instructions} works, and append the default only when the caller did not replace the template. A full custom template should be the whole prompt.
  2. Let callers replace the disfluency sentence without replacing the template. True can keep today's English line. A string should replace that line. False can keep the opt-out, but that opt-out string needs the same override, or a string should be the only way to name the fillers.

The bool should keep doing what it does now for vocabulary filtering. On Fish Audio, disfluencies=True also adds English few-shots (Um, uh... really?) inside llm_instructions(). Those already drop out if the template omits {tts.markup.llm_instructions}. If a custom disfluency string is set, those few-shots should not keep teaching um and uh.

The same hook should cover the other hardcoded steering prose. When nonverbal steering removes some sounds, _sound_guidance appends an English Non-verbal sounds: line. pace appends Keep a {pace} overall speaking pace. A caller who replaces the steering fragment should be able to replace those too.

English defaults should stay as they are when no override is set.

Workarounds / Alternatives

tts_instructions_append can contradict the English line. The English line is still in the prompt, and the model often follows um, uh anyway.

Replacing tts_instructions_template, including omitting {tts.markup.llm_instructions}, does not stop the append. resolve_expressive_options concatenates steering_instructions() after the template whenever disfluencies is not None.

Omitting speech_steering does not help. The default sets disfluencies=True, and resolve merges that default.

disfluencies=False still injects English: No fillers (um, uh). Sound composed and fluent.

Additional Context

Seen on livekit-agents 1.8.2. The same strings are in steering_instructions() on main.

Reproduction:

  1. AgentSession with inference.TTS (xAI tts-1) and expressive set.
  2. tts_instructions_template is a Japanese guide. It names えっと、あの、まあ、そうですね and does not include {tts.markup.llm_instructions}.
  3. speech_steering={"disfluencies": True, ...}.
  4. Render with resolve_expressive_options(..., provider_key="xai", default=DEFAULT_EXPRESSIVE_OPTIONS), then Instructions.render.
  5. The result still contains fillers (um, uh) and openers (oh, well, so).

Docs describe disfluencies as a bool for "um" and "uh", on by default: Expressive mode. Marker labels should stay English. convert_markup only recognizes that vocabulary. This request is only about the prose that tells the LLM which spoken fillers to use.

Additional Context

No response

主要言語
Python
スター
14.6k
フォーク
3.9k
平均マージ
2日 23時間
マージ済み PR（30日）
258

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

livekit/agents のほかの issue

livekit/agents の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。