vllm-project/vllm-omni

/v1/realtime has no tool/function-calling support

Open

#5,522 opened on Jul 29, 2026

View on GitHub
 (2 comments) (1 reaction) (1 assignee)Python (1,067 forks)github user discovery
enhancementgood first issuehelp wanted

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

Environment: vllm-omni 0.24.0, serving Qwen/Qwen3-Omni-30B-A3B-Instruct via vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --no-async-chunk.

What we're trying to do: Use the self-hosted /v1/realtime websocket as a full speech-to-speech "brain" for a LiveKit Agents voice worker, including tool/function calling (the model deciding to call e.g. a weather-lookup or inventory-check tool mid-conversation) — the same pattern OpenAI's and Gemini's realtime APIs support.

What we found: The realtime protocol has no wire mechanism for tools at all, at either the vLLM or vllm-omni layer:

  • SessionUpdate (vllm/entrypoints/speech_to_text/realtime/protocol.py) only declares a model field.
  • RealtimeConnection.handle_event (vllm/entrypoints/speech_to_text/realtime/connection.py) only reads event.get("model") from session.update — nothing else is parsed or threaded into generation.
  • vllm-omni's RealtimeConnection override (vllm_omni/entrypoints/openai/realtime_connection.py) only adds audio-delta emission on top of that; it doesn't add tool support either.
  • There's no server→client event for a function-call request, and no client→server event to submit a tool result.

Impact: A realtime session can never be given tool definitions, so the model has no way to know it has any tools available — it just verbally declines ("I can't provide real-time weather updates...") rather than invoking anything, even though the same checkpoint supports tool-calling fine via /v1/chat/completions with tools=[...].

Request: Add tool/function-calling support to the realtime protocol — e.g. a tools field on SessionUpdate, plus request/response event types for function calls (mirroring OpenAI's response.function_call_arguments.* / conversation.item.create shape, or vLLM's own tool_parsers used for chat completions). Happy to help test against a patch if useful.

Contributor guide