Gemini 3.8 Live is async-by-default: plugin ends up in a hybrid blocking/non-blocking state, behavior/scheduling cannot be set per tool, and tool results wait for playout

Đang mở
  • Có gì không ổn?
#7,302 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
35/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
audio-video-rtc, backend-api-design

Hướng nghiên cứu

Start with the cited paths: livekit/agents/llm/_provider_format/google.py and livekit/plugins/google/realtime/realtime_api.py for effective tool behavior, then llm/tool_context.py and livekit/plugins/google/utils.py for per-tool settings. For result timing, read voice/agent_activity.py and voice/tool_executor.py alongside the open #6741. The issue describes three related outcomes: correct async defaults and silent scheduling, per-tool behavior/scheduling, and sending results without waiting for playout.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

gemini-3.8-live (GA, 2026-09-15) makes asynchronous function calling the default: "Async execution (behavior: NON_BLOCKING) is now the default function calling mode. You can still use synchronous blocking mode for backwards compatibility by setting behavior: BLOCKING on your tool declarations" (model page, "Migrating from Gemini 3.1 Flash Live"). gemini-3.8-live-extended-thinking accepts only NON_BLOCKING. Three things in livekit-agents / livekit-plugins-google (main fc22671, identical to 1.8.2 for the files below) get in the way of using that properly. Measured on 1.8.1 with the production-size system instruction of a phone agent, 15 live trials per arm.

1. With the new default, the plugin ends up in a hybrid state it does not know about

tool_behavior is not set → to_fnc_ctx omits behavior from every FunctionDeclaration (livekit-agents/livekit/agents/llm/_provider_format/google.py:152, :167: if tool_behavior is not None: schema["behavior"] = ...). On 3.8 the server then treats every tool as NON_BLOCKING, but supports_silent_scheduling is derived from what the client set (livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py:708-711), so it stays False: a tool result with reply_required=False (typical for a fire-and-forget update, or for a tool that completed inside an interrupted turn, voice/generation.py marks those) is sent without scheduling: SILENT, the model answers anyway, and the warning at realtime_api.py:718-723 ("a tool result wants no reply … declare the tools NON_BLOCKING on the Gemini API to keep it silent") fires on every call: 15/15 and 15/15 in two arms with the default, 10/10 with tool_behavior=BLOCKING set explicitly, 0/15 with tool_behavior=NON_BLOCKING set explicitly. Suggestion: on models where the documented default is NON_BLOCKING, either send behavior explicitly (whatever the user chose, defaulting to the model's default) or derive supports_silent_scheduling from the effective behavior, not from the option.

2. behavior and scheduling are per session, they need to be per tool

RealtimeModel(tool_behavior=…, tool_response_scheduling=…) writes the same behavior on every FunctionDeclaration (_provider_format/google.py:152-168) and the same scheduling on every FunctionResponse (livekit-plugins-google/livekit/plugins/google/utils.py:62-110). A real agent needs both kinds at once: a CRM update or a "mark call successful" is NON_BLOCKING + SILENT (nothing to say about it), a calendar lookup whose result is the next sentence is BLOCKING, a hangup must be BLOCKING because with NON_BLOCKING the model may keep talking after calling it and the speech is cut by the hangup. FunctionToolInfo / RawFunctionToolInfo (llm/tool_context.py:184-214) carry no such field. #2367 asked for exactly @function_tool(behavior="NON_BLOCKING") and was closed as completed by #3482, which shipped only the session-wide option. A plugin-level map like the one #7197 just gave the Phonic plugin (configs_for_tools, tool name → config) would be enough and would not touch the core.

3. The FunctionResponse is held until all queued speech has been played

voice/agent_activity.py:4667-4676 (while self._current_speech or self._speech_q, comment "most realtime models don't support generating multiple responses at the same time") sends the tool result only after playout. For an async tool this defeats the point: scheduling: INTERRUPT can never interrupt anything, WHEN_IDLE arrives late, and the model's own decision to keep talking while the tool runs is followed by a result that lands after the talking is over. The AsyncToolset path has the same wait (voice/tool_executor.py:541-550, wait_for_idle() before generate_reply). #6741 (open) touches the adjacent problem of keeping the generation open after a NON_BLOCKING call; the send-after-playout is untouched. On 3.8, where the model is designed around this, the result should be sent as soon as the tool returns, with the scheduling telling the model what to do with it.

Why it matters, numbers

Same agent, same script that asks the model to persist a value mid-call, gemini-3.8-live, 15 trials per arm: with NON_BLOCKING + WHEN_IDLE the model performs the mid-call save in 7/15 trials with a median of 5.1 s from the end of the caller's sentence to the toolCall and 0 unanswered caller turns; with BLOCKING set explicitly 3/10 saves and 16.8 s; on gemini-3.1-flash-live-preview (sync only) 4/10 and 19.2 s. Async is the better mode on 3.8 — but today it can only be turned on for all tools at once, including hangup_call and the end-of-call finalization tool, and its result still waits for playout.

Environment

livekit-agents 1.8.1, livekit-plugins-google 1.8.1, google-genai 2.22.0, Python 3.12, models gemini-3.8-live, gemini-3.8-live-extended-thinking, gemini-3.1-flash-live-preview (API key). Line numbers above are from main at fc22671. Related: google-gemini/gemini-live-api-examples#46 (the model-side repetition before a function call, measured with the same harness).

Ngôn ngữ chính
Python
Star
14.6k
Fork
3.9k
Merge trung bình
2 ngày 21 giờ
Pull request đã merge (30 ngày)
249

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của livekit/agents

Tất cả issue của livekit/agents

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.