[Bug]: Qwen3-Omni seed-TTS — model speaks the task description / answers the prompt instead of reading it; ~2x worse on the vLLM 0.27 rebase

Summary

The nightly Omni · Accuracy Test (test_qwen3_omni_seed_tts_wer_bench) fails on dev/vllm-align (the vLLM 0.27 rebase, PR #5976) with mean WER ~0.38–0.51 against the 0.35 gate, while main scores 0.27–0.29.

I reproduced the gap locally as a controlled A/B and inspected every synthesized utterance. The failures are not audio corruption — Qwen3-Omni stops behaving as a TTS engine for a subset of prompts and instead speaks the task description, or answers the reference sentence, or babbles. Because the WER used by the gate is uncapped, a few such clips dominate the mean.

Importantly, this failure mode already exists on main; the rebase roughly doubles its frequency. So there are two problems: a latent instruction-following bug (product) and a real regression on the rebase branch (blocking PR #5976).

Reproduction (local, controlled A/B)

Same box (2 GPUs), same dataset (zhaochenyang20/seed-tts-eval, en, 1088 utterances), same harness invocation, same deploy config vllm_omni/deploy/qwen3_omni_moe.yaml, --seed-tts-wer-save-items + SEED_TTS_WER_SAVE_AUDIO_DIR so every utterance is scored and kept:

stack	vLLM / torch	mean WER	median	clips with WER>1.0
origin/main	0.26.0 / 2.11	0.3047	0.0000	89
dev/vllm-align @ 25b80bb9	0.27.0 / 2.13	0.3821	0.0000	151

Per-utterance comparison over the same 1088 prompts:

• 80 of main's 89 broken clips are broken on dev too (a stable hard-clip set)
• 71 clips regress (fine on main → broken on dev); only 9 improve — an 8:1 asymmetry, not a symmetric numerics reshuffle
• net +0.0774 mean WER per clip

CI shows the same direction and a slightly larger gap (main 0.2665/0.2773/0.2788/0.2932 vs dev 0.3756/0.3826/0.3916/0.5114), so this is not specific to my hardware (I ran on L20X; CI uses H100).

What the model actually does

Examples where main speaks the reference verbatim and dev does not (same prompt, same seed-tts request):

reference (must be spoken verbatim)	main	dev
Appearances are deceptive.	"Appearances are deceptive."	"Nezha de Siajin's speech in the same language and style as the reference audio"
Don't take any chances.	"Don't take any chances."	"Get decisive speech in the same style as the reference audio. Calm, measured, and slightly cautionary tone. Don't take any chances."
He always stutters when he's nervous, that's why he delegated the speech.	verbatim	"Not cause things flit in prose again, at the false as when the distairs, Miss Jew-tation the back-shadow…" (babble)

Longer failures are the model answering the sentence instead of reading it:

• ref I feel sick, what should I do? → spoken: "I'm sorry to hear that you're feeling unwell. Here are some general steps you can take. 1. Rest… 2. Stay hydrated…" (97 s of audio)
• ref Please search The Best of Steely Dan: Then and Now saga. → spoken: "Sure. Here is the text you requested spoken in the style of the reference audio. The Best of Steely Dan… is a compilation album released by Steely Dan in 1993…"

Failure-mode breakdown (clips with WER>0.5, share of total error):

mode	main	dev
over-generation (spoke ≫ reference: answers/preambles)	59 clips, 72% of error	97 clips, 64% of error
garbled (similar length, wrong words)	46 clips, 17% of error	97 clips, 28% of error
clips that literally speak the phrase "reference audio" (task text leaking into speech)	64	146

References that are questions are hit hardest on dev: mean WER 2.53 (60% above 1.0) vs 0.33 for statements.

Why the metric amplifies this

• WER is uncapped. _jiwer_wer (vllm_omni/benchmarks/data_modules/seed_tts_eval.py) returns (S+D+I)/N over reference words. A 97 s spoken answer against a 7-word reference scores ~7.4. One such clip moves a 1088-clip mean by ~0.007; the top 10 clips alone contribute 0.055 of dev's 0.382, and clips with WER>1.0 carry 82% of the total.
• The talker is unseeded and stochastic (qwen3_omni_moe.yaml: stage 1 temperature: 0.9, top_k: 50; the bench passes no seed), so the tail — and therefore the gated mean — is noisy run to run.

A tail-insensitive statistic (per-clip WER capped at 1.0, or a "fraction of clips with WER>0.5" gate) would make this job diagnose content failures instead of tracking a handful of runaway clips.

Ruled out

• Harness / measurement. git diff origin/main..HEAD touches exactly one bench file (vllm_omni/benchmarks/patch/patch.py), and only with an inactive probe_request_rate port (default 0.0, never passed by CI) and a defensive getattr. The seed-tts dataset module, prompt, eval code, and tests/e2e/accuracy/qwen3_omni/ are unchanged; total_input_tokens is bit-identical (158002) across branches, proving the same prompts.
• Sampler precision. vLLM 0.27 drops dtype=torch.float32 from the top-p softmax in topk_topp_sampler.py, but Sampler.forward already upcasts logits to fp32 beforehand in both versions, so it is a no-op.
• Whisper grader drift. Same decode path in both runs; a direct A/B moved WER by −0.007.
• Total audio volume is not a reliable signal: locally both stacks produced ~6320 s and main had more >20 s clips (17 vs 14), even though CI showed dev consistently longer. The robust cross-environment signal is content mismatch, not duration.

Remaining suspects (for the rebase delta)

The thinker decodes greedily (stage 0 temperature: 0.0), so the flip from "read the text" to "describe/answer the text" is a deterministic text-path perturbation. Consistent with this, Daily-Omni MCQ moves bit-stably 71.60% → 70.68% (11 single-token flips) at the same boundary.

1. torch 2.13 compiled-region behavior in the thinker — this branch already carries three torch-2.13 inductor workarounds (acf245b2, e3b42850, de3502cf).
2. e3efeaaf ("return a bare tensor from the qwen3-omni thinker when no capture is requested") — changing a support_torch_compiled forward's return arity can move the hidden-state capture relative to the compiled region; the tensors handed to the talker were never verified bit-identical.

Note the "good" dev nightlies 2942/2944/2946 actually ran a main-lineage commit (900a7f081), so no dev build before 2949 was ever benchmarked — the entire migration (vLLM 0.26→0.27, torch 2.11→2.13, ~30 rebase commits) lands in one unseparated window and nothing has been bisected yet.

Suggested next steps

1. Bit-compare the thinker hidden states handed to the talker between the two stacks for one failing utterance (isolates suspect 2 from suspect 1).
2. Re-run the dev stack with enforce_eager on stage 0; if the extra failures disappear, it is inductor/torch 2.13.
3. Independently of the root cause: cap per-clip WER (or gate on the fraction of clips above a threshold) and seed the talker, so this gate stops being dominated by a handful of stochastic runaways.
4. Treat the task-text leakage ("…speech in the same style as the reference audio" spoken aloud, 64 clips even on main) as a product bug in its own right.

Related: #5480 (same job, earlier era where main itself sat at 0.40 with median 0.23 — distinct from this, where both branches have median 0.00 and differ only in tail mass).