SpeakingRateStream blocks the event loop: per-frame Python STFT loop in `_stft` / `_spectral_flux`

已关闭
  • 哪里不对？
#7,902 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

@omerekin 已经在做这个了。

开始于 2026年10月17日。

  • #7903 来自 @omerekin —— 未关闭
  • #7938 来自 @longcw —— 未关闭

评估

难度
3/5
预计耗时
1-2 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
停滞
技术栈
python
领域
audio-video-rtc, performance

调研方向

Locate SpeakingRateStream._stft and _spectral_flux, then trace audio draining through _main_task. Run the existing speaking-rate tests and event-loop timing checks, comparing spectral-flux output and loop-blocking measurements before and after the change. Done means the calculation is vectorized without changing results and queued TTS audio no longer causes the measured stalls.

由索引模型根据 Issue 内容生成。

描述

TranscriptSynchronizer always runs SpeakingRateDetector on the agent's TTS audio, on the job's event loop. SpeakingRateStream._stft computes the STFT one frame at a time in a Python loop (about 80 np.fft.rfft calls per 1 s window at 24 kHz), and _spectral_flux loops over the frames again. Because RoomIO's _ParticipantAudioOutput.capture_frame queues TTS audio without pacing, _main_task drains a whole lump of a reply (10 windows per second of audio) without yielding.

Measured on livekit-agents 1.8.2, numpy 2.5.2, Apple M3 (the code is unchanged in 1.8.8, 2.0.0b1 and main):

  • 0.61 ms per window; _main_task draining 20 s of queued 24 kHz audio holds the loop ~123 ms (2 s: 7.6 ms, 5 s: 26 ms, 10 s: 58 ms).
  • In a local test conversation driven by synthetic speech, with LIVEKIT_AGENTS_LOOP_BLOCK_WARN_MS=50: 4-5 loop blocks of 59-131 ms per run, a few seconds into each agent turn. With the 200 ms output buffer, stalls that size can be audible, and on smaller cloud machines they'd be roughly twice as long.

The same computation vectorized gives a bit-identical result (480 random cases across 6 sample rates and 4 lengths) and is ~3.8x faster (0.16 ms per window; 20 s of queued audio holds the loop ~38 ms). It replaces _stft + _spectral_flux:

def _spectral_flux(
    self, audio: np.ndarray[tuple[int], np.dtype[np.float32]], sample_rate: int
) -> float:
    frame_length = int(sample_rate * 0.025)  # 25ms
    hop_length = frame_length // 2  # 50% overlap
    num_frames = (len(audio) - frame_length) // hop_length + 1
    if num_frames < 2:
        return 0.0
    window = np.hanning(frame_length)
    scale_factor = 1.0 / np.sqrt(np.sum(window**2))
    frames = np.lib.stride_tricks.sliding_window_view(audio, frame_length)[::hop_length]
    magnitudes = np.abs(np.fft.rfft(frames * window, axis=1) * scale_factor)
    return float(np.mean(np.sum(np.abs(np.diff(magnitudes, axis=0)), axis=1)))

It might also be worth yielding (await asyncio.sleep(0)) between windows in _main_task, so a long lump never holds the loop, whatever the per-window cost.

Happy to open a PR with this if it's useful.

主要语言
Python
星标
14.6k
派生
3.9k
平均合并
3 天
30 天内合并 PR
262

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

livekit/agents 的其他 Issue

查看 livekit/agents 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。