[Bug]: MiniCPM-o 4.5: streaming chat TTS conditions Talker on 1 text token → unintelligible audio / Seed-TTS WER ≫ 1

Summary

When serving MiniCPM-o 4.5 with vLLM-Omni via OpenAI-compatible /v1/chat/completions (modalities: text+audio, chat_template_kwargs.use_tts_template=true):

• stream=false: Thinker→Talker bridge passes the full tts_bos..tts_eos text region (e.g. 9 tokens for a short English sentence). Speech is conditioned correctly.
• stream=true (default for openai-chat-omni / Seed-TTS bench): Talker almost always logs generating speech for 1 tokens, while tts_hidden_states is still ~10 frames long. emb_text and projected hiddens broadcast-mismatch, producing garbage audio.

Seed-TTS WER then looks catastrophic even though:

• HTTP / PCM capture / ASR all succeed (Request failed=0, No PCM=0, ASR/WER failed=0)
• Returned text is complete and correct

This is a serving / stage-bridge bug under streaming, not a MiniCPM-o model-quality or Whisper-eval artifact.

Environment

Item	Value
Model	openbmb/MiniCPM-o-4_5 (local checkpoint)
vLLM	0.25.0
vLLM-Omni	0.25.0rc2.dev47+g10d3a2591.d20260720 (git cfd7a079e)
Endpoint	POST /v1/chat/completions (openai-chat-omni)
Deploy	vllm_omni/deploy/minicpmo_4_5.yaml (async_chunk: false, stage0 Thinker + stage1 Talker)
Bridge	vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py → llm2tts
Talker	vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_tts.py
Client / bench	vllm-omni bench serve --dataset-name seed-tts --backend openai-chat-omni (payload sets "stream": true by default)
Extra body	{"chat_template_kwargs":{"use_tts_template":true}}

Reproduce

1. Serve

vllm serve /path/to/MiniCPM-o-4_5 --omni --port 28889 \
  --trust-remote-code \
  --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml

2. Minimal A/B (same prompt, stream on/off)

Prompt: Get the trust fund to the bank early.

import requests

url = "http://127.0.0.1:28889/v1/chat/completions"
base = {
    "model": "/path/to/MiniCPM-o-4_5",
    "messages": [{"role": "user", "content": "Get the trust fund to the bank early."}],
    "modalities": ["text", "audio"],
    "max_tokens": 32,
    "temperature": 0,
    "chat_template_kwargs": {"use_tts_template": True},
}

for stream in (False, True):
    r = requests.post(url, json={**base, "stream": stream}, timeout=180, stream=stream)
    # drain stream if needed; then inspect stage-1 server logs

3. Observe stage-1 (Talker) logs

Request	Log
stream=false	4.5 Talker: generating speech for **9** tokens / inputs_embeds shape=[1, **11**, 768]
stream=true	4.5 Talker: generating speech for **1** tokens / inputs_embeds shape=[1, **12**, 768]

Healthy shape is num_text + 2 (text_eos + audio_bos).
Broken shape 12 with text_tokens=1 implies ~10 hidden frames broadcast against 1 token id.

With return_token_ids=true (non-stream or aggregated stream deltas), Thinker still emits the full sequence:

[1949, 279, 6950, 3802, 311, 279, 6073, 4124, 13, 151704, 151645]
→ "Get the trust fund to the bank early.<|tts_eos|><|im_end|>"

So the LLM side is fine; only the payload handed to Talker is wrong under streaming.

4. Seed-TTS symptom (streaming bench)

vllm-omni bench serve \
  --omni --port 28889 \
  --dataset-name seed-tts \
  --dataset-path /path/to/seed-tts-eval \
  --seed-tts-locale en \
  --num-prompts 32 \
  --backend openai-chat-omni \
  --endpoint /v1/chat/completions \
  --extra-body '{"chat_template_kwargs":{"use_tts_template":true}}' \
  --seed-tts-wer-save-items \
  --trust-remote-code \
  --model /path/to/MiniCPM-o-4_5

Observed (32 prompts, streaming default):

===== Seed-TTS eval (seed-tts-eval protocol) =====
Evaluated (WER, lower is better):        32
Mean WER:                                2.6943
Median WER:                              1.0000
Request failed:                          0
No PCM captured:                         0
ASR / WER failed:                        0
==================================================

ASR transcripts of the synthesized WAV are gibberish (letter salad / repetition), while message.content matches the prompt sentence.

Root cause analysis

Bridge slice assumes token_ids length ≡ hidden length

In llm2tts:

end_idx = tts_eos_idx if tts_eos_idx is not None else full_hidden.shape[0]
tts_token_ids_slice = torch.tensor(full_token_ids[tts_bos_idx:end_idx], dtype=torch.long)
tts_hidden_slice = full_hidden[tts_bos_idx:end_idx]

Under streaming, the thinker output reaching the bridge often has:

• truncated output.token_ids (missing most of the sentence / missing <|tts_eos|> = 151704)
• longer latent / hidden_states (prompt + generated, or otherwise not length-aligned with the truncated ids)

When <|tts_eos|> is absent, end_idx = full_hidden.shape[0]. Then:

Slice	Behavior	Length (example)
full_token_ids[bos:end]	Python list clamp	1
full_hidden[bos:end]	Tensor index	~10–11

Talker silently broadcasts the mismatch

llm_embeds = tts.emb_text(tts_token_ids)          # [1, H]
hidden_embeds = tts.projector_semantic(tts_hidden)  # [10, H]
tts_embeds = llm_embeds + hidden_embeds             # broadcast → [10, H]
# + text_eos + audio_bos → inputs_embeds [1, 12, 768]

Conditioning is garbage → unintelligible waveform → Seed-TTS WER ≫ 1.

Note: llm2tts currently does del streaming_context and does not implement a Qwen3-style chunked / wait-for-eos streaming bridge.

Actual vs Expected

	Text	Audio / Talker condition
Actual (stream=true)	Complete sentence	Talker conditioned on 1 text token + misaligned hiddens → garbage speech
Actual (stream=false)	Complete sentence	Talker conditioned on full tts_bos..tts_eos region (e.g. 9 tokens)
Expected	Complete sentence	Same full TTS region for streaming and non-streaming; lengths of tts_token_ids and tts_hidden_states must match

Suggested fix directions

1. Streaming transfer contract: only call llm2tts / submit Talker when Thinker finished and output.token_ids contains the full generated sequence (including <|tts_eos|>), not a delta / truncated view.
2. Hard alignment in llm2tts: refuse or clamp with end = min(end_idx, len(full_token_ids), full_hidden.shape[0]) and assert len(tts_token_ids) == tts_hidden.shape[0]; log + fail loud on mismatch instead of broadcasting.
3. Prefer token-driven end: if <|tts_eos|> is missing, do not extend the token slice with full_hidden.shape[0] unless tokens and hiddens are known to be aligned; optionally wait for eos / final update.
4. Tests: unit test for “truncated token_ids + longer hidden” (today’s happy-path tests always keep equal lengths). Add an online smoke: same prompt stream vs non-stream must yield the same Talker text_tokens count.
5. Bench workaround (does not fix product streaming): vllm-omni bench serve ... --no-stream for MiniCPM-o Seed-TTS WER until (1) lands.

Impact

• Severity: High — any streaming voice chat / Seed-TTS / realtime demo for MiniCPM-o 4.5 produces wrong speech while text looks fine.
• Scope: Confirmed on MiniCPM-o 4.5 Thinker→Talker (llm2tts). Other omni TTS stacks with streaming bridges may need a similar audit; Qwen3-Omni already has an explicit streaming chunk path that MiniCPM-o 4.5 lacks.
• Misleading metrics: Seed-TTS Mean WER ~2.7 with zero transport errors looks like a model failure; it is a bridge bug under stream=true.

Additional context

• Prerequisite: without use_tts_template=true, audio can be near-silence (~0.5 s) and WER collapses for a different reason (empty TTS region). This issue is after that flag is set correctly.
• Related symptom family: text complete vs speech wrong/truncated (see also long-form Chinese audio tail truncation). This ticket is specifically the streaming 1-token conditioning / hidden mismatch failure mode.