Expressive Mode: `speech_steering` always injects an English disfluency line, and that text cannot be customized

Abierto
  • ¿Algo no está bien?
#7,554 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@rrfunde ya está trabajando en esto.

Desde el 14/10/2026.

  • #7824 de @rrfunde — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
54/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
ai-infra-agents

Línea de trabajo

Start with steering_instructions() and resolve_expressive_options in livekit/agents/tts/_provider_format.py, then trace how AgentActivity._inject_expressive_instructions adds the rendered text to chat context. Check the existing expressive-mode tests and run the relevant tests before changing behavior. Done means callers can customize steering prose and disfluency text while English defaults and vocabulary filtering remain unchanged.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement
Feature Type

Would make my life easier

Feature Description

tts_instructions_template and tts_instructions_append can replace or extend the expressive markup guide. The delivery line that speech_steering injects cannot. It is hardcoded English, and resolve_expressive_options appends it after any custom template.

steering_instructions() in livekit/agents/tts/_provider_format.py renders disfluencies as one of two fixed sentences:

  • True: Sprinkle in natural fillers (um, uh) and openers (oh, well, so), zero to two per turn, never mechanical.
  • False: No fillers (um, uh). Sound composed and fluent.

DEFAULT_EXPRESSIVE_OPTIONS sets disfluencies=True. resolve_expressive_options merges that default, so the English line is injected whenever expressive mode is on, including expressive=True and a custom tts_instructions_template that never mentions fillers. Omitting {tts.markup.llm_instructions} skips the provider markup block. It does not skip this line. AgentActivity._inject_expressive_instructions then puts the rendered text in the LLM chat context.

That breaks non-English agents. A Japanese agent that wants えっと、あの、まあ、そうですね still sees um, uh and oh, well, so in the same prompt. Models follow the English examples. Setting disfluencies=False does not help if you still want fillers. It only swaps in the other English sentence.

Please make every expressive prompt fragment customizable, the same way the markup guide already is. The disfluency line is the one we need to edit. A language-appropriate filler list should be able to replace um, uh without rewriting LiveKit's tag vocabulary.

Two changes would cover it:

  1. Stop appending steering_instructions() unconditionally. Expose that fragment as a placeholder, the way {tts.markup.llm_instructions} works, and append the default only when the caller did not replace the template. A full custom template should be the whole prompt.
  2. Let callers replace the disfluency sentence without replacing the template. True can keep today's English line. A string should replace that line. False can keep the opt-out, but that opt-out string needs the same override, or a string should be the only way to name the fillers.

The bool should keep doing what it does now for vocabulary filtering. On Fish Audio, disfluencies=True also adds English few-shots (Um, uh... really?) inside llm_instructions(). Those already drop out if the template omits {tts.markup.llm_instructions}. If a custom disfluency string is set, those few-shots should not keep teaching um and uh.

The same hook should cover the other hardcoded steering prose. When nonverbal steering removes some sounds, _sound_guidance appends an English Non-verbal sounds: line. pace appends Keep a {pace} overall speaking pace. A caller who replaces the steering fragment should be able to replace those too.

English defaults should stay as they are when no override is set.

Workarounds / Alternatives

tts_instructions_append can contradict the English line. The English line is still in the prompt, and the model often follows um, uh anyway.

Replacing tts_instructions_template, including omitting {tts.markup.llm_instructions}, does not stop the append. resolve_expressive_options concatenates steering_instructions() after the template whenever disfluencies is not None.

Omitting speech_steering does not help. The default sets disfluencies=True, and resolve merges that default.

disfluencies=False still injects English: No fillers (um, uh). Sound composed and fluent.

Additional Context

Seen on livekit-agents 1.8.2. The same strings are in steering_instructions() on main.

Reproduction:

  1. AgentSession with inference.TTS (xAI tts-1) and expressive set.
  2. tts_instructions_template is a Japanese guide. It names えっと、あの、まあ、そうですね and does not include {tts.markup.llm_instructions}.
  3. speech_steering={"disfluencies": True, ...}.
  4. Render with resolve_expressive_options(..., provider_key="xai", default=DEFAULT_EXPRESSIVE_OPTIONS), then Instructions.render.
  5. The result still contains fillers (um, uh) and openers (oh, well, so).

Docs describe disfluencies as a bool for "um" and "uh", on by default: Expressive mode. Marker labels should stay English. convert_markup only recognizes that vocabulary. This request is only about the prose that tells the LLM which spoken fillers to use.

Additional Context

No response

Lenguaje dominante
Python
Estrellas
14.6k
Forks
3.9k
Merge medio
3 d
PR fusionados (30 d)
262

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de livekit/agents

Todos los issues de livekit/agents

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.