OpenAI realtime: a text user turn queued during a tool-output push deletes the function_call_output from the server conversation

クローズ
  • 何か問題がありますか？
#7,880 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@Epsilon006 がすでに取り組んでいます。

2026年10月17日 から。

  • #7904 @Rehansanjay による — マージされずにクローズ
  • #7905 @Epsilon006 による — マージ済み

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
ai, backend

調査の方向性

Read voice/agent_activity.py around lines 4156–4160, 4737, and 4809–4853, then trace llm/_realtime/openai.py around lines 1385–1386 and 1498–1527 and the chat context diff. Start by running the reported scripted fake websocket reproduction with delayed acknowledgements. Done means concurrent text and tool output pushes preserve the function_call_output in the server conversation; cover both output push paths with a regression test.

索引モデルが issue の本文から書いたものです。

説明

Bug Description

With the OpenAI realtime model, a text user turn (session.generate_reply(user_input=...) or session.run(user_input=...)) that is queued while a tool output is being pushed can delete that tool output from the server-side conversation. The agent's local chat_ctx still contains the output, so local and remote contexts diverge, and every later response, including the tool reply itself, is generated without the tool result. With an open function_call and no output, the model typically says the operation is "still running".

The sequence, with line numbers from livekit-agents 1.8.8. The code is the same in 1.8.0, where we first saw it, and in 2.0.0b1.

  1. A function tool returns while another speech is queued. The output push waits for the speech queue to drain (voice/agent_activity.py:4809-4819). It then copies the remote mirror, appends the outputs and calls update_chat_ctx (agent_activity.py:4850-4853). Inside RealtimeSession.update_chat_ctx, that call holds _update_chat_ctx_lock until the server acknowledges the conversation.item.create (llm/_realtime/openai.py:1498-1527).
  2. Meanwhile the text-input branch runs (agent_activity.py:4156-4160). It does chat_ctx = self._rt_session.chat_ctx.copy(), which reads the remote mirror (_remote_chat_ctx, openai.py:1385-1386) before the lock is held, so the tool output is not in it yet. It adds the user message, then waits on the same lock inside update_chat_ctx.
  3. The output's ack lands, and the remote mirror now contains the function_call_output. The second update_chat_ctx then diffs its stale copy against the mirror (_create_update_chat_ctx_events → compute_chat_ctx_diff). The output appears in to_remove, so a conversation.item.delete is sent for it.

The push of interrupted tool outputs (agent_activity.py:4737) also copies before taking the lock, so any two concurrent pushes can presumably drop each other's items the same way.

The race depends on timing: it needs the server's ack to arrive after the second push has taken its copy. With realistic network latency that is the common case. With an instant ack in a fake server it doesn't reproduce.

Expected Behavior

A text user turn adds the user message without removing items another push added in the meantime. The function_call_output stays in the server conversation, and the tool reply and later responses see it.

Reproduction Steps
# Reproduced offline: a pytest using a scripted fake realtime websocket server
# (the server keeps the conversation the way the API does and acks each event after ~50 ms).
# No OpenAI calls are involved.
1. AgentSession with openai.realtime.RealtimeModel (modalities=["text"]) and an Agent with one function tool.
2. Have a reply call the tool while another speech handle is still queued, so the output push waits (agent_activity.py:4811).
3. While that push awaits its ack, queue a text user turn: session.run(user_input="...") or generate_reply(user_input="...").
4. Watch the client events: after the conversation.item.create for the function_call_output is acked,
   the user-turn push sends conversation.item.delete for that same function_call_output item.
5. Every later response.create is generated from a server conversation with the function_call but no output.
# With an immediate ack (no latency), step 4 does not happen.
# We can share the test, or turn it into a regression test in this repo, if that helps.

Possible fixes:

  • Build the context to push under the lock, from the mirror as it stands at that moment. For example, update_chat_ctx could accept a function that edits the current remote context inside the lock.
  • Or have the text-input path append its one message with a single conversation.item.create (previous_item_id = the last remote item) instead of pushing a full-context diff.

Either way, the tool-output pushes at agent_activity.py:4737 and :4850 should get the same treatment.

Operating System

macOS 15 (offline reproduction); Linux containers in production

Models Used

OpenAI Realtime via livekit-plugins-openai

Package Versions
livekit-agents==1.8.0
livekit-plugins-openai==1.8.0
# the relevant code is unchanged in livekit-agents 1.8.8 and 2.0.0b1
Session/Room/Call IDs

None: reproduced offline against a scripted server.

主要言語
Python
スター
14.6k
フォーク
3.9k
平均マージ
3日
マージ済み PR（30日）
262

環境構築

このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

livekit/agents のほかの issue

livekit/agents の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。