Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

elevenlabs: text-to-dialogue (v3/v4) alignment timestamps are chunk-relative, breaking synced transcripts

Abierto
  • ¿Algo no está bien?
#7,773 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@satheeshk50 ya está trabajando en esto.

Desde el 13/10/2026.

  • #1 de @satheeshk50 — fusionado
  • #7784 de @satheeshk50 — abierto

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
58/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
audio-video-rtc

Línea de trabajo

Read the ElevenLabs plugin’s _DialogueConnection._recv_loop and _accumulate_alignment path, introduced in #6838, and compare it with the regular text-to-speech websocket handling. Reproduce with eleven_v4_turbo and sync_alignment=True, checking how alignment behaves across audio chunks, including chunks without alignment. Done means timed words increase monotonically across the utterance and match the audio, as they do for eleven_flash_v2_5.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Bug Description

For text-to-dialogue models (eleven_v3*, eleven_v4*), the ElevenLabs plugin emits word timings that are wrong when sync_alignment=True. The timings restart near 0 on every audio chunk instead of increasing across the utterance.

The text-to-dialogue websocket returns alignment.char_start_times_ms relative to each chunk. The regular text-to-speech websocket returns them relative to the start of the context. _DialogueConnection._recv_loop passes the chunk-relative values straight into _accumulate_alignment, which treats them as relative to the context. This path was introduced in #6838.

As a result, with use_tts_aligned_transcript=True and sync_transcription=True, the timed transcript for a 13s utterance spans only 0–0.8s. Timestamps go backwards, and some words have end_time < start_time. Synced captions don't track the audio.

Expected Behavior

Timed words should increase monotonically across the utterance and line up with the audio, as they do for eleven_flash_v2_5.

Reproduction Steps
1. Set ELEVEN_API_KEY and run the script below.
2. Compare the flash and v4 output.

import asyncio, sys, aiohttp
from livekit.plugins import elevenlabs
from livekit.agents.tts.tts import USERDATA_TIMED_TRANSCRIPT

TEXT = ["Thanks, that's helpful. ", "Dr. Smith said it cost $1,250 on 3/4/2025, around 5pm. ",
        "Can you walk me through what happened next? ", "And how did that make you feel at the time?"]

async def run(model):
    async with aiohttp.ClientSession() as http:
        tts = elevenlabs.TTS(model=model, language="en", encoding="pcm_24000", http_session=http)
        stream = tts.stream()
        for t in TEXT:
            stream.push_text(t)
        stream.end_input()
        audio_s, words = 0.0, []
        async for ev in stream:
            for w in ev.frame.userdata.get(USERDATA_TIMED_TRANSCRIPT, []) or []:
                words.append((audio_s, w))
            audio_s += ev.frame.duration
        await stream.aclose(); await tts.aclose()
    print(f"=== {model}: audio {audio_s:.2f}s")
    for at, w in words:
        print(f"  {w.start_time:6.2f}-{w.end_time:5.2f}s  (audio pos {at:5.2f}s)  {str(w)!r}")

for m in sys.argv[1:]:
    asyncio.run(run(m))

# python repro.py eleven_flash_v2_5 eleven_v4_turbo

Abridged output:

=== eleven_flash_v2_5: audio 9.35s      # monotonic, matches audio
    0.02- 0.46s  (audio pos  0.00s)  'Thanks, '
    4.90- 5.41s  (audio pos  5.29s)  '5pm.  '
    8.94- 9.38s  (audio pos  9.29s)  'time? '

=== eleven_v4_turbo: audio 13.28s       # resets every chunk; max start_time 0.80s, 10 backward jumps
    0.00- 0.32s  (audio pos  0.49s)  'Thanks, '
    0.48- 0.00s  (audio pos  1.69s)  'helpful. '
    0.16- 0.00s  (audio pos  8.49s)  '5pm. '
    0.24- 0.72s  (audio pos 13.09s)  'time? '

Raw websocket messages confirm the API behavior. On text-to-speech/.../multi-stream-input, charStartTimesMs keeps increasing across messages (0 … 499, 534 … 2159, …). On text-to-dialogue/multi-stream-input, every message's char_start_times_ms starts at 0. The dialogue endpoint also sends some audio messages with no alignment.

Operating System

macOS 15

Models Used

ElevenLabs eleven_v4_turbo (also affects eleven_v3 / eleven_v3_conversational via the same path)

Package Versions
livekit-agents==1.8.4
livekit-plugins-elevenlabs==1.8.4
# also present in 1.8.6 / main (a7c62e30)
Session/Room/Call IDs

No response

Proposed Solution
# In _DialogueConnection, keep a per-context audio offset and add it to each chunk's
# alignment before calling _accumulate_alignment. Base the offset on the decoded audio
# duration pushed so far, not on the previous chunk's last char end, because some audio
# chunks carry no alignment. With PCM: offset_ms += len(pcm_bytes) / (2 * sample_rate) * 1000.
# Pipecat's ElevenLabsDialogueTTSService applies a similar running offset,
# advancing by each chunk's alignment span.
Additional Context

Pipecat handles this in its dialogue service: "Alignment times restart at zero in every message, so the clock advances by each chunk's span."

The ElevenLabs docs describe alignment as timing data "for the chunk" but don't specify the time base.

Screenshots and Recordings

No response

Lenguaje dominante
Python
Estrellas
14.6k
Forks
3.9k
Merge medio
2 d 23 h
PR fusionados (30 d)
177

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de livekit/agents

Todos los issues de livekit/agents

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.