vllm-project/vllm-omni

/v1/realtime has no tool/function-calling support

Aperta

#5522 aperta il 29 lug 2026

 (2 commenti) (1 reazione) (1 assegnatario)Python (1067 fork)github user discovery
enhancementgood first issuehelp wanted

Metriche repository

Star
 (4990 stelle)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

Environment: vllm-omni 0.24.0, serving Qwen/Qwen3-Omni-30B-A3B-Instruct via vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --no-async-chunk.

What we're trying to do: Use the self-hosted /v1/realtime websocket as a full speech-to-speech "brain" for a LiveKit Agents voice worker, including tool/function calling (the model deciding to call e.g. a weather-lookup or inventory-check tool mid-conversation) — the same pattern OpenAI's and Gemini's realtime APIs support.

What we found: The realtime protocol has no wire mechanism for tools at all, at either the vLLM or vllm-omni layer:

  • SessionUpdate (vllm/entrypoints/speech_to_text/realtime/protocol.py) only declares a model field.
  • RealtimeConnection.handle_event (vllm/entrypoints/speech_to_text/realtime/connection.py) only reads event.get("model") from session.update — nothing else is parsed or threaded into generation.
  • vllm-omni's RealtimeConnection override (vllm_omni/entrypoints/openai/realtime_connection.py) only adds audio-delta emission on top of that; it doesn't add tool support either.
  • There's no server→client event for a function-call request, and no client→server event to submit a tool result.

Impact: A realtime session can never be given tool definitions, so the model has no way to know it has any tools available — it just verbally declines ("I can't provide real-time weather updates...") rather than invoking anything, even though the same checkpoint supports tool-calling fine via /v1/chat/completions with tools=[...].

Request: Add tool/function-calling support to the realtime protocol — e.g. a tools field on SessionUpdate, plus request/response event types for function calls (mirroring OpenAI's response.function_call_arguments.* / conversation.item.create shape, or vLLM's own tool_parsers used for chat completions). Happy to help test against a patch if useful.

Guida contributor