[Feature]: Add word-level timestamps to TTS streaming output via shared forced aligner
#3,631 opened on May 15, 2026
Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
🚀 The feature, motivation and pitch
Voice-agent use cases (barge-in, mid-utterance interrupt) need word-level timestamps aligned to the streamed audio. Currently the vllm-omni TTS streaming path only returns audio chunks, so an orchestrator can't tell which word the user interrupted on. This applies to every TTS model in the repo: Qwen3-TTS, VoxCPM2, CosyVoice3, FishSpeech, OmniVoice, MOSS-TTS-Nano, Voxtral-TTS.
Proposal: emit (word, start_ms, end_ms) alongside each audio chunk in the shared TTS streaming response by default, computed by a model-agnostic forced aligner that takes (audio_chunk, text, sr) → list[WordTimestamp].
Suggested scope:
- New shared utility
vllm_omni/utils/forced_aligner.pythat lazy-loads a forced aligner and exposesalign(audio_chunk, text, sr) -> list[WordTimestamp]. Stateless, reusable across all TTS backends. - Extend the shared streaming response schema in
vllm_omni/entrypoints/openai/protocol/audio.pywith aword_timestampsfield. - Hook into the shared streaming path
vllm_omni/entrypoints/openai/serving_speech_stream.pyso every TTS model benefits without per-model changes. - Offline example under
examples/offline_inference/qwen3_tts/demonstrating the timestamps. - E2e test in
tests/e2e/online_serving/covering at least Qwen3-TTS and one other model (e.g. VoxCPM2 or CosyVoice3) to prove the cross-model contract.
Marked as good first issue because it's purely additive: no AR / vocoder / scheduler changes — one new utility + one response-schema field + one shared streaming hook.
Alternatives
- Client-side alignment. Caller runs a forced aligner after receiving full audio. Rejected: requires every caller to load a second model and breaks barge-in scenarios that need timestamps before the utterance finishes.
- Per-model forced aligners. Wire it into each TTS model's own streaming code. Rejected: duplicates work across 7+ models and drifts over time. Keeping the aligner model-agnostic is the right boundary.
Additional context
- Shared streaming path:
vllm_omni/entrypoints/openai/serving_speech_stream.py,serving_speech.py,protocol/audio.py. - TTS models that will all benefit:
vllm_omni/model_executor/models/{qwen3_tts,voxcpm2,cosyvoice3,fish_speech,omnivoice,moss_tts_nano,voxtral_tts}/.