[RFC]: Qwen3-TTS high-throughput hardening

Motivation.

Streaming Qwen3-TTS at c ≥ 64 currently produces audible audio gaps for every user. Our primary business workload is Qwen3-TTS-Base + voice_clone, streamed via OpenAI /v1/audio/speech. SLO target is max_underrun_ms_p95 ≤ 100 ms; every cell currently misses this by 50–200×.

Bench setup: H20, bench_tts_client_continuity.py per-chunk underrun tracking, default vs 2-GPU split (s0=128, s1=1, devices: 0/1).

Insights (data behind the workstreams):

insight	data	implication
Continuity fails by 50–200× SLO across every cell	Base voice_clone undr_p99: 16.2 / 17.7 / 19.1 s at c=64/128/256. CV: 8.6 / 8.6 / 8.4 s. SLO is 100 ms	Throughput fixes alone cannot meet SLO; scheduler RFC needed in parallel
2-GPU split helps voice_clone, hurts CV at c ≥ 128	Base c=128 undr_p99: 17.7 → 16.2 s (better). CV c=128: 8.6 → 11.6 s (worse). CV c=256: 8.4 → 12.0 s (worse)	Single-axis "split stages" is not enough; Stage 1 becomes new bottleneck for CV at high c
voice_clone's Stage 0 is 11× slower than CV's	chunk-emit interval (per @ischencheng profile): voice_clone 263 ms, CV 23 ms	voice_clone is Stage-0-bound; CV is Stage-1-bound. Optimizations are not symmetric
Stage 1 default max_num_seqs: 1→10 already gives 2–3× TTFA at c=4/8	PR #3322 ablation by @ischencheng	yaml-level batching works; structural batched forward (chunked_decode with bs>1) contributes ~0% — confirmed
enable_prefix_caching=true crashes Qwen3-Omni serving	Issue #3510 reproducer	Ref-audio KV reuse via vLLM prefix cache is blocked until #3510 is fixed
Stage 1 scheduler is per-chunk round-robin	Codex review of omni_generation_scheduler.py:71-72 + chunk_transfer_adapter.py:407-428	128 streams interleave through Stage 1 chunk-by-chunk; out-of-scope for this RFC, scheduler RFC will address

Scope: high-throughput (c ≥ 32) deployment, voice_clone primary. Out of scope: scheduler / SLA work, low-latency mode, MIG/MPS, async D2H pipeline (#3509) — each gets its own RFC.

Proposed Change.

Five parallel workstreams. SLO-level continuity (100 ms underrun) requires a separate scheduler RFC.

Tasks

WS	task	owner	scope	acceptance	justification
WS-1	AR / talker optimization	@Sy0307	Land 1–2 PRs porting #3221's talker patterns (graph-safe sampling primitives, @support_torch_compile evaluation, kernel tuning) into existing vllm_omni/.../qwen3_tts/ talker. No new model arch	Base voice_clone Stage 0 TTFT or chunk-emit interval improves ≥ 15% at c=4/8	voice_clone Stage-0-bound (263 ms vs 23 ms); #3221 has working thinker patterns
WS-2	Prefix / speaker cache for ref-audio KV reuse	@JuanPZuluaga	Primary: debug #3510, get enable_prefix_caching=true working on omni path. Fallback: extend SpeakerCacheManager (#2630) to be KV-aware (get_kv/put_kv/invalidate, LRU eviction)	At ref_audio_hash reuse ≥ 80%, Base voice_clone TTFT_p50 at c=8–32 improves ≥ 30%	voice_clone is Stage-0-bound; ref-audio prefill is the heavy step; reuse eliminates it on cache hit
WS-3	Prefilling optimization (cache-miss path)	@R2-Y	Pick 1–2: cross-request batched ref-audio prefill / chunked prefill for ref-audio segment / FP8 ref-audio encoder	Cache-miss Base voice_clone TTFT improves ≥ 20% over main	voice_clone Stage-0-bound; WS-2 covers cache-hit, WS-3 covers cache-miss
WS-4	Adaptive producer-consumer (credit-based)	@linyueqian	Per-stream chunk credits between Stage 0 and Stage 1. Stage 0 scheduler skips credit=0 streams in LM step. Static credit count first; adaptive is follow-up	SHM queue depth bounded; Stage 0 idle on backpressured streams decreases vs baseline	At c=128 with 1 Stage 1 replica, producer/consumer rates are mismatched; current adapter queue is unbounded
WS-5	Code2wav batch + multi-replica	@ischencheng	Build on PR #3322. Add multi-replica Stage 1 via num_replicas (framework merged). Verify sticky per-stream routing in SHM connector. Publish (num_replicas, s1) config table	At c=128/256, 2× Stage 1 replicas improve RTF ≥ 30% over 1-replica, no routing bugs	CV becomes Stage-1-bound at c ≥ 128 under 2-GPU split (insight #2); horizontal Stage 1 scaling is the only path

Dependencies

WS-1 (talker)        — independent
WS-5 (multi-replica) — independent
WS-2 (cache)         — independent (primary path = #3510 debug; fallback = self-managed)
   └─→ WS-3 (cache-miss path)
                                ┌─ WS-5 throughput data
WS-4 (adaptive)        ←────────┤
                                └─ WS-2 cache hit/miss state

WS-1 / WS-2 / WS-5 startable immediately. WS-3 / WS-4 have soft dependencies (can default while starting).

Aggregate acceptance

metric	Base voice_clone c=128	CustomVoice c=128
max_underrun_ms_p95	substantially improved (< 2 s; 100 ms SLO needs scheduler RFC)	same
TTFT_p99 / TTFT_p50	≤ 3.0	≤ 3.0
RTF_p50	not worse than current best by > 5%	same
success_rate	100%	100%
consolidated deliverable	vllm_omni/deploy/qwen3_tts_high_throughput.yaml published with WS-5 recommended config

This RFC does not commit to meeting the 100 ms underrun SLO. Scheduler RFC handles that.

Out of scope

topic	reason	future RFC
Scheduler / SLA (Stage 1 bounded-K active-stream window, admission control)	Required for 100 ms SLO but is structural change to OmniGenerationScheduler	yes
Low-latency mode (c ≤ 4)	Different operating point; NVIDIA #3221 single-stage + TensorRT is the natural fit	yes
MIG / MPS GPU partitioning	Infrastructure-side, separate concern	yes
Async D2H + stage pipeline transfer	@tzhouam owns via #3509	cross-ref only

Risks

R1	WS-5 multi-replica may surface SHM connector cross-replica routing bugs. Sticky per-stream routing must be verified before claiming throughput win
R2	WS-2 primary path depends on #3510 being tractable. If structural, fallback (self-managed SpeakerCacheManager extension) adds engineering cost without changing user-visible behavior
R3	WS-4 credit-based flow control may oscillate under variable load. Static credit first; adaptive is follow-up
R4	Scheduler is out of scope. voice_clone at c ≥ 128 will still produce audible gaps after this RFC. Manage expectations explicitly

Related in-flight work

ref	who	relation
PR #3485	merged	initial_codec_chunk_frames=1 — first-chunk TTFP fix, already in main
PR #3322 / Issue #3163	@ischencheng	Single-GPU code2wav batched decoding. WS-5 extends to multi-replica
PR #3221	@vklimkov-nvidia	Single-stage talker + TensorRT vocoder. WS-1 borrows thinker patterns; does not merge
Issue #3510	@amy-why-3459	enable_prefix_caching=true causes Qwen3-Omni requests to hang. WS-2 addresses this
RFC #3509	@tzhouam	Async D2H + stage pipeline transfer. Not pursued in this RFC; cross-referenced

(Bench charts and per-chunk underrun visualizations will be attached as images below.)

Feedback Period.

One week (until 2026-05-19). Each WS owner please confirm the assignment or propose adjustments by then.

CC List.

@Sy0307 @JuanPZuluaga @R2-Y @ischencheng @tzhouam @gcanlin @hsliuustc0106