Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Run the live interviewer on a local speech pipeline

オープン
  • 何か問題がありますか？
#330 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@alanhc がすでに取り組んでいます。

2026年10月17日 から。

評価

この issue はまだ評価されていません。

説明

On an RTX 5070 Ti, a local speech pipeline behind Gemini Live's BidiGenerateContent socket has run whole CodeTrial interviews with tool calls, barge-in, LiveKit and a browser, and once it prepares the reply during the silence window, its first audio arrives a median 0 ms after that window ends. Gemini Live, measured from the same server, arrives a median 0.5 s after it. #105 closed point 3 on the M1 Pro figure of 4.83 s, which had none of these conditions; this issue is the measurement under them that the closing comment asked for.

Setup
Server Intel Core i7-14700, 64 GB RAM, RTX 5070 Ti 16 GB (driver 580.173), Ubuntu 24.04
CodeTrial #110 plus one commit that opens CODETRIAL_GEMINI_LIVE_URL instead of Google's socket when it is set (18 lines in src/gemini.rs); unset, nothing changes
Live interviewer a shim answering BidiGenerateContent: Silero VAD on the CPU, faster-whisper 1.2.1 large-v3-turbo (int8_float16), gemma-4-12b-it Q4_K_M on llama.cpp llama-server with thinking off, Kokoro 1.0 on onnxruntime-gpu 1.30
Reports and text calls scripts/gemini-shim.py --thinking off from #110, in front of the same llama-server
LiveKit self-hosted livekit-server 1.13.8 in --dev mode; during these runs it still handed clients Google's and Twilio's STUN servers, since fixed in its config
GPU memory gemma 8.2 GB, the speech models 2.4 GB, and 4.0 GB for an unrelated process

Against the conditions #105 named

Condition Result
Tool calls 11-minute soak through scripts/browser-check.sh (BROWSER_CHECK_AGENT=dispatch), "I'm stuck, can I get a hint?" every 35 s: 18 of 18 requests heard, 18 log_hint calls, 32 interviewer turns, none failed
Barge-in 11-minute soak, the candidate speaking every 12 s: 55 of 55 interruptions sent as interrupted, each followed by a normal turn; the history keeps only the sentences that had started playing
A browser and LiveKit in the path both soaks and both hand runs below went through CodeTrial, the browser and LiveKit
Cold prefill first sentence 2.1 s with the 4,759-token instruction uncached, against 51 s on the M1 Pro; prefilling when the session opens would hide it, not yet tried
Resumption not implemented; the soak ends on "no transport restart", which only Gemini's ten-minute socket cap causes

First audio after the silence window

The window is DEFAULT_GEMINI_SILENCE_MS (3,000 ms), which Gemini Live waits too. From 600 ms into the silence the shim transcribes, asks the model and synthesizes the reply without sending it, and sends it when the window ends; speech before then discards it. A tool call is decided in advance but only made when the window ends. A hint request is not prepared at all, since its reply depends on the log_hint result; hint turns took 1.9 to 2.9 s after the window in the tool-call soak.

11-minute soak turns median p90 max failures
reply started when the window ends 56 1,078 ms 1,424 ms 2,232 ms 0
reply prepared during the window, same request every 12 s 56 0 ms 0 ms 1,609 ms 0
reply prepared during the window, five different lines 52 0 ms 1,140 ms 1,589 ms 0

Gemini Live, with the gemini_bench.py from #105 run from this server, took a median 3.50 s from end of speech to first audio with the same window, so about 0.5 s after it; five of six turns fell between 3.29 and 3.58 s, and one took 47.0 s. From the M1 Pro it was 3.49 s (3.23 to 3.67). Both runs send no tools and synthesized speech, and the soaks above are a different harness, so the two are compared on the same span, not in one experiment.

Human speech

Two interviews by hand with a microphone, about five and seven minutes, Two Sum, everything local; #110 has both runs. The second recorded all six REACTO steps and three hints, and the report came back on the first call in 17.6 s. These were not timed for first audio.

What does not work yet

  • gemma-4-12b repeats itself when the candidate repeats a request: asked for a hint 30 times after the ladder is spent, 49% of its sentences nearly repeat an earlier one. Qwen3.5-9B repeated less but named the technique unasked and ended an interview early, Qwen3-14B repeated as much and did not fit beside the speech models, and DRY sampling turned repeats into paraphrases.
  • No model tried calls log_hint once the ladder is spent, so the shim recognizes a hint request in the transcript and makes the call itself.
  • whisper once produced a sentence the candidate did not say ("I'm going to go to the next slide."), and the interviewer answered it.
  • The phase judge recorded steps with the wrong quote, and the interviewer's record_framework_evidence used a confidence of 5 on a 0 to 100 scale.
  • The candidate speech in the soaks is synthesized; only the two hand runs were a person.

Code

The speech shim and the measurements are in alanhc/local-voice-bench (live_shim.py; the README has every stage and how to run it). The CodeTrial commit is b587f21 on alanhc/codetrial local-voice, on top of #110.

Decision needed

Is a local live interviewer something CodeTrial should allow at all? If so, the smallest step is CODETRIAL_GEMINI_LIVE_URL in tree and the speech shim out of tree, so the open problems above stay with the shim and the Rust side keeps one provider. If not, this issue records the measurement and can be closed.

主要言語
Rust
スター
160
フォーク
64
平均マージ
1日 12時間
マージ済み PR（30日）
124

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

sysprog21/codetrial のほかの issue

sysprog21/codetrial の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。