Audio routing freeze mid-call (1.6.9)

Start by tracing AudioRecognition audio routing and the STT stream lifecycle during multi-turn conversations, as identified in the issue. Compare what happens during successful turns with the freeze, especially around activity handoffs; the report confirms room audio frames continue arriving while STT/VAD receive none. Done means identifying and fixing the shared routing failure so user-state events and STT output continue across turns.

Description

Audio routing stops working mid-call. user_state stays stuck on "listening", no user_state_changed events are emitted, STT produces no output, and the call hangs up after inactivity prompts.

User continues speaking throughout (confirmed via separate audio stream subscription), but the audio doesn't reach STT/VAD/turn detection components.

Happens randomly during multi-turn conversations - not always on the same turn. Can't reproduce reliably but occurs frequently enough to block production deployment.

Expected vs Actual

Expected: Turn detection works every turn, user_state transitions normally, STT produces transcripts

Actual: Audio routing freezes, user_state stuck on "listening", no user_state_changed events, no STT output

Environment

• Version: livekit-agents 1.6.9, livekit 1.1.14
• Deployment: Self-hosted, outbound SIP
• Models: Deepgram nova-3 (STT), gpt-4.6-luna (LLM), Deepgram aura-2 (TTS)

Comprehensive Testing

Tested multiple configurations to isolate the issue. All configurations exhibit the same freeze behavior.

Configuration 1: Silero VAD + Turn Detector v1-mini

vad=inference.VAD(model="silero", min_speech_duration=0.05, min_silence_duration=0.25, activation_threshold=0.4)
turn_handling=TurnHandlingOptions(
    turn_detection=inference.TurnDetector(version="v1-mini"),
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call

Configuration 2: No VAD + Turn Detector v1-mini

vad=None  # Explicitly disabled
turn_handling=TurnHandlingOptions(
    turn_detection=inference.TurnDetector(version="v1-mini"),
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call
Evidence: Logs confirm Session VAD: None

Configuration 3: No VAD + STT Endpointing

vad=None
turn_handling=TurnHandlingOptions(
    turn_detection="stt",  # Deepgram handles endpointing
    endpointing={"min_delay": 0.5, "max_delay": 1},
    interruption={"mode": "vad", "min_words": 3},
)

Result: Freezes mid-call

What This Proves

The bug is NOT caused by:

• Silero VAD specifically (tested with vad=None)
• Turn Detector v1-mini (tested with turn_detection="stt")
• Corrupted VAD/STT instance state (tested with fresh instances per turn)
• Network/SIP issues (confirmed audio frames arrive via separate subscription)

The bug appears to be a framework-level issue in:

• Audio routing in AudioRecognition
• Audio frame delivery to STT/VAD/turn detection components
• Affects all turn detection methods equally

Additional Testing and Workarounds (All Failed)

Configuration changes tested:

• Removed allow_interruptions=False from all session.say() / generate_reply() calls, replaced with temporary elevation of interruption.min_words threshold - no change
• Switched from deprecated silero.VAD.load() to inference.VAD(model="silero") - no change
• Switched from text-based turn_detector.multilingual.MultilingualModel to audio-based inference.TurnDetector(version="v1-mini") - no change
• Created fresh VAD and STT instances per session (not shared across sessions) - still freezes

Runtime workarounds tested:

1. Runtime VAD refresh - agent.update_options(vad=fresh_vad) when freeze detected - still freezes
2. Override _update_vad() - Force fresh VAD instances on every activity - still freezes, logging shows VAD channel exists (closed=False) but emits no events
3. Disable STT pipeline reuse - Force fresh STT WebSocket each time - still freezes

Evidence During Freeze

When freeze occurs:

• user_state = "listening" (stuck)
• agent_state = "listening"
• No user_state_changed events emitted
• No STT output/transcripts
• Audio frames arrive at room (verified via rtc.AudioStream.from_participant())
• VAD channel exists and closed=False (when VAD is used)

Example Timeline

Successful turns followed by freeze (common pattern)

16:04:51 - Agent speaks (turn 1)
16:04:53 - User response detected successfully
16:04:57 - Agent speaks (turn 2)
16:05:00 - User response detected successfully
16:05:03 - Agent speaks (turn 3)
16:05:06 - User response detected successfully
16:05:09 - Agent speaks (turn 4)
16:05:34 - Agent waits for user response
          [FREEZE OCCURS - audio routing stops]
          User continues speaking (confirmed via separate audio stream)
          No user_state_changed events emitted
          No STT transcription output
16:05:39 - WARNING: No user_state events for 5.0s agent_state=listening user_state=listening
16:05:41 - Agent: inactivity prompt 1
16:05:46 - WARNING: No user_state events for 5.0s
16:05:48 - Agent: inactivity prompt 2
16:05:53 - WARNING: No user_state events for 5.0s
16:05:55 - Agent: inactivity prompt 3
16:06:00 - WARNING: No user_state events for 5.0s
16:06:09 - Agent: farewell message and hangup

Note: The freeze can occur after any number of successful turns (sometimes turn 2, sometimes turn 5+). No consistent pattern to when it occurs.

Impact

Blocking production deployment. High call failure rate, poor user experience. Affects self-hosted deployments with outbound SIP.

Root Cause Analysis Needed

Audio frames reach the room but don't reach STT/VAD/turn detection. Need framework investigation of:

1. AudioRecognition audio routing - How frames are routed to detection components during multi-turn conversations
2. STT stream lifecycle - Why STT stops consuming frames mid-call
3. Activity transitions - Whether audio routing breaks during activity handoffs

The fact that all turn detection methods fail identically suggests the issue is in the shared audio routing infrastructure, not in any specific detection component.