STT turn detection: stopped_speaking_at ignores word timestamps, and is wrong after a handoff or a gap in mic audio

Start from the repro script (repro_stopped_speaking_at.py) and run all five scenarios to capture the baseline, then read the three cited sites: audio_recognition.py where _last_speaking_time is set from the final transcript and overwritten by END_OF_SPEECH (L1298/L1356), agent.py's start_time_offset (L541) versus _process_stt_event adding _input_started_at (agent_activity.py L1191), and _compute_end_of_turn_metrics (L109) which drops metrics after a gap. Done means all scenarios report stopped_speaking_at within a few ms of the true last-word end on every stream (start, during/after handoff, after a 5s input gap), with no turn-commit/endpointing timing change. Check _last_speaking_time's second use at L1718 so the metric and the endpointing wait stay separate.

Bug Description

With STT turn detection, stopped_speaking_at on user messages doesn't match when the user actually stopped talking, and neither do transcription_delay, end_of_turn_delay or the agent's e2e_latency. The causes are in the core turn handling, not in a particular STT plugin. We noticed it with Soniox, but the repro below uses a fake STT.

I found three causes (line numbers are from 1.8.3; the same code is on main):

1. END_OF_SPEECH overwrites the word timestamp. The final transcript sets _last_speaking_time from its end_time (audio_recognition.py L1298). But Soniox, Deepgram and AssemblyAI send END_OF_SPEECH with no alternatives and no speech_end_time, so L1356 sets it to time.time() instead. stopped_speaking_at ends up being the time END_OF_SPEECH arrived, which is late by the provider's endpointing delay (about 0.6s with Soniox), and transcription_delay is always 0. This is the case #4388 reported; the fix there only helps when END_OF_SPEECH carries end_time. If the user's mic goes quiet right after they speak (e.g. the track is muted), END_OF_SPEECH only arrives once audio resumes, so the value is off by the whole pause.

2. STT streams started later in the session use the wrong time base (any plugin). A new pipeline gets created after the session started, for example on every agent handoff when the agent overrides stt_node (the pipeline is only reused for the default node, agent_activity.py L1076). The default stt_node sets start_time_offset relative to the session/recorder start, because the new pipeline has no _input_started_at yet (agent.py L541). _process_stt_event then also adds the pipeline's _input_started_at (L1191), so the time since session start is counted twice. The result is in the future and gets clamped to now. Even if the provider puts a timestamp on END_OF_SPEECH, every stream after the first one reports the arrival time.

3. Gaps in the mic audio aren't accounted for (any plugin). Provider time only moves forward while audio is being pushed. After 5s with no frames from the mic, end_time + _input_started_at is 5s too early, which is before the turn's started_speaking_at. _compute_end_of_turn_metrics (L109) then drops all four metrics for every turn after the gap.

Expected Behavior

stopped_speaking_at should be the wall-clock time the user's last word ended, as reported by the provider. That should hold on every STT stream in the session, including after a gap in the mic audio. e2e_latency should then be the time from that point to the agent's first audio.

Reproduction Steps

1. Get the script: repro_stopped_speaking_at.py. It doesn't need API keys or a LiveKit server.
   • A fake mic pushes 20ms frames in real time and tags each one, so the script knows when every frame was captured.
   • A fake streaming STT sends events the way most streaming plugins do: the final arrives 0.3s after the last word with end_time = start_time_offset + audio position, and END_OF_SPEECH follows 0.6s after the last word.
2. pip install livekit-agents==1.8.3
3. Run:

python repro_stopped_speaking_at.py plain
python repro_stopped_speaking_at.py plain --eos-timestamp      # END_OF_SPEECH also carries the word end
python repro_stopped_speaking_at.py task --eos-timestamp       # an AgentTask runs, then the parent agent resumes
python repro_stopped_speaking_at.py input_gap --eos-timestamp  # the mic sends nothing for 5s
python repro_stopped_speaking_at.py input_gap


There's one line per user turn. The last line of a run can be a turn that was committed when the session closed, before its END_OF_SPEECH arrived (+0.003s and +0.302s below).


scenario=plain eos_timestamp=False
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.603 s
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.603 s
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.603 s
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.600 s
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.601 s
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.003 s
scenario=plain eos_timestamp=True
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.004 s
(5 more lines, all +0.004 s)
scenario=task eos_timestamp=True
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.003 s
during task                  stream start_time_offset  3.009 s   stopped_speaking_at - true last-word end: +0.603 s
after task                   stream start_time_offset  6.012 s   stopped_speaking_at - true last-word end: +0.602 s
after task                   stream start_time_offset  6.012 s   stopped_speaking_at - true last-word end: +0.602 s
after task                   stream start_time_offset  6.012 s   stopped_speaking_at - true last-word end: +0.602 s
after task                   stream start_time_offset  6.012 s   stopped_speaking_at - true last-word end: +0.302 s
scenario=input_gap eos_timestamp=True
start                        stream start_time_offset  0.003 s   stopped_speaking_at - true last-word end: +0.005 s
after 5 s with no mic audio  stream start_time_offset  0.003 s   stopped_speaking_at - true last-word end: +0.005 s
after 5 s with no mic audio  stream start_time_offset  0.003 s   stopped_speaking_at - true last-word end: missing
after 5 s with no mic audio  stream start_time_offset  0.003 s   stopped_speaking_at - true last-word end: missing
scenario=input_gap eos_timestamp=False
start                        stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.603 s
after 5 s with no mic audio  stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +5.601 s
after 5 s with no mic audio  stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.602 s
after 5 s with no mic audio  stream start_time_offset  0.002 s   stopped_speaking_at - true last-word end: +0.602 s

In the input_gap runs, the first "after" turn is the one whose last word came just before the gap.

Operating System

macOS 26.4.1 (repro). Production is Linux.

Models Used

Not model-specific: the repro uses a fake STT and LLM. We first saw it in production with Soniox stt-rt-v5 and STT turn detection.

Package Versions

livekit==1.1.18
livekit-agents==1.8.3
livekit-api==1.2.0
python 3.14.7

Session/Room/Call IDs

n/a, reproduces locally.

Proposed Solution

1. If END_OF_SPEECH has no timestamp, keep the `_last_speaking_time` set by the turn's last final/preflight, and only fall back to `time.time()` if there wasn't one.
2. Use the same time base on both sides. Either subtract the startup offset that `stt_node` added before adding `_input_started_at`, or compute the offset from the pipeline's `_input_started_at` once the first frame has been pushed.
3. For gaps, track how late each frame reaches the STT stream relative to `_input_started_at`, and add the smallest lag of the frames from the word onward. Keep negative values: they mean `_input_started_at` itself was stamped late. That happens when the event loop is busy while a new pipeline receives its first frames, and the queued audio then arrives in a burst. This is roughly what we run in our agent:


from bisect import bisect_left


class StreamClock:
    def __init__(self, input_started_at: float) -> None:
        self.input_started_at = input_started_at
        self.pushed = 0.0  # seconds of audio pushed so far
        self.lag_floor: list[tuple[float, float]] = []  # (audio position, lag), lag increasing

    def on_frame(self, duration: float, arrived_at: float) -> None:
        self.pushed += duration
        lag = arrived_at - self.input_started_at - self.pushed
        while self.lag_floor and self.lag_floor[-1][1] >= lag:
            self.lag_floor.pop()
        self.lag_floor.append((self.pushed, lag))

    def wall_time(self, audio_position: float) -> float:
        if not self.lag_floor:
            return self.input_started_at + audio_position
        i = bisect_left(self.lag_floor, audio_position - 1e-6, key=lambda entry: entry[0])
        return self.input_started_at + audio_position + self.lag_floor[min(i, len(self.lag_floor) - 1)][1]

Additional Context

Our agents override stt_node, and some of our sessions mute the user's mic for a while, so we hit all three. For now we compute turn latency ourselves from the provider word timestamps.

_last_speaking_time also sets the endpointing wait (extra_sleep = endpointing_delay + last_speaking_time - now, L1718). So just keeping the word end in case 1 would make STT-mode turns commit earlier, about 0.6s earlier with Soniox. The metric probably needs its own value, so that commit timing stays the same. Along the same lines, for plugins that do send speech_end_time, cases 2 and 3 can already shift when a turn commits. I haven't measured that.

On #4388, @longcw asked whether end_time is relative to when the STT connection was created or to the pushed audio duration. Cases 2 and 3 are that question: it's the pushed audio, starting at the stream's start_time_offset, so wall time has to account for when the audio arrived.

Related: #4388, #6093.

Screenshots and Recordings

n/a