vllm-project/vllm-omni

/v1/realtime has no tool/function-calling support

オープン

#5,522 opened on 2026/07/29

 (2 件のコメント) (1 件のリアクション) (1 人の担当者)Python (1,067 件のフォーク)github user discovery
enhancementgood first issuehelp wanted

Repository metrics

Stars
 (4,990 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Environment: vllm-omni 0.24.0, serving Qwen/Qwen3-Omni-30B-A3B-Instruct via vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --no-async-chunk.

What we're trying to do: Use the self-hosted /v1/realtime websocket as a full speech-to-speech "brain" for a LiveKit Agents voice worker, including tool/function calling (the model deciding to call e.g. a weather-lookup or inventory-check tool mid-conversation) — the same pattern OpenAI's and Gemini's realtime APIs support.

What we found: The realtime protocol has no wire mechanism for tools at all, at either the vLLM or vllm-omni layer:

  • SessionUpdate (vllm/entrypoints/speech_to_text/realtime/protocol.py) only declares a model field.
  • RealtimeConnection.handle_event (vllm/entrypoints/speech_to_text/realtime/connection.py) only reads event.get("model") from session.update — nothing else is parsed or threaded into generation.
  • vllm-omni's RealtimeConnection override (vllm_omni/entrypoints/openai/realtime_connection.py) only adds audio-delta emission on top of that; it doesn't add tool support either.
  • There's no server→client event for a function-call request, and no client→server event to submit a tool result.

Impact: A realtime session can never be given tool definitions, so the model has no way to know it has any tools available — it just verbally declines ("I can't provide real-time weather updates...") rather than invoking anything, even though the same checkpoint supports tool-calling fine via /v1/chat/completions with tools=[...].

Request: Add tool/function-calling support to the realtime protocol — e.g. a tools field on SessionUpdate, plus request/response event types for function calls (mirroring OpenAI's response.function_call_arguments.* / conversation.item.create shape, or vLLM's own tool_parsers used for chat completions). Happy to help test against a patch if useful.

コントリビューターガイド