vllm-project/vllm-omni

[Feature]: Add word-level timestamps to TTS streaming output via shared forced aligner

Open

#3,631 opened on May 15, 2026

View on GitHub
 (15 comments) (0 reactions) (2 assignees)Python (1,067 forks)github user discovery
enhancementgood first issuetts

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

🚀 The feature, motivation and pitch

Voice-agent use cases (barge-in, mid-utterance interrupt) need word-level timestamps aligned to the streamed audio. Currently the vllm-omni TTS streaming path only returns audio chunks, so an orchestrator can't tell which word the user interrupted on. This applies to every TTS model in the repo: Qwen3-TTS, VoxCPM2, CosyVoice3, FishSpeech, OmniVoice, MOSS-TTS-Nano, Voxtral-TTS.

Proposal: emit (word, start_ms, end_ms) alongside each audio chunk in the shared TTS streaming response by default, computed by a model-agnostic forced aligner that takes (audio_chunk, text, sr) → list[WordTimestamp].

Suggested scope:

  • New shared utility vllm_omni/utils/forced_aligner.py that lazy-loads a forced aligner and exposes align(audio_chunk, text, sr) -> list[WordTimestamp]. Stateless, reusable across all TTS backends.
  • Extend the shared streaming response schema in vllm_omni/entrypoints/openai/protocol/audio.py with a word_timestamps field.
  • Hook into the shared streaming path vllm_omni/entrypoints/openai/serving_speech_stream.py so every TTS model benefits without per-model changes.
  • Offline example under examples/offline_inference/qwen3_tts/ demonstrating the timestamps.
  • E2e test in tests/e2e/online_serving/ covering at least Qwen3-TTS and one other model (e.g. VoxCPM2 or CosyVoice3) to prove the cross-model contract.

Marked as good first issue because it's purely additive: no AR / vocoder / scheduler changes — one new utility + one response-schema field + one shared streaming hook.

Alternatives

  • Client-side alignment. Caller runs a forced aligner after receiving full audio. Rejected: requires every caller to load a second model and breaks barge-in scenarios that need timestamps before the utterance finishes.
  • Per-model forced aligners. Wire it into each TTS model's own streaming code. Rejected: duplicates work across 7+ models and drifts over time. Keeping the aligner model-agnostic is the right boundary.

Additional context

  • Shared streaming path: vllm_omni/entrypoints/openai/serving_speech_stream.py, serving_speech.py, protocol/audio.py.
  • TTS models that will all benefit: vllm_omni/model_executor/models/{qwen3_tts,voxcpm2,cosyvoice3,fish_speech,omnivoice,moss_tts_nano,voxtral_tts}/.

Contributor guide