Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Agent state stays "speaking" after a realtime reply in sessions without audio output

Open Beginner friendly
  • Is something wrong?
#7,866 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
ai-infra-agents

Research direction

The bug is in AgentActivity._realtime_generation_task_impl, where the transition back to listening sits inside the if audio_output is not None: branch. Compare it with the pipeline LLM path, which resets the state without an audio output, and check where _update_agent_state is called. Done means a text-only realtime reply ends in listening (or thinking when background speeches remain), with a test that covers a session without audio output.

Written by the indexing model from the issue text.

Description

Bug Description

In a session without audio output, a realtime model's reply leaves the agent state at speaking. It never goes back to listening. Clients that take turns on lk.agent.state hang after the first realtime reply. Our text-mode interviews and our simulation harness are both affected.

In AgentActivity._realtime_generation_task_impl, the transition back to listening sits inside if audio_output is not None:. A text-only session has no audio output, so it never takes that branch. The pipeline (STT-LLM-TTS) path resets the state without an audio output, so the two paths behave differently.

Expected Behavior

After a realtime generation finishes, the agent state goes to listening, or to thinking while background speeches remain, whether or not the session has an audio output. This matches the pipeline LLM path.

Reproduction Steps
1. Set LIVEKIT_API_KEY and LIVEKIT_API_SECRET and run the script below. No room is needed.
2. Compare the pipeline LLM with the two realtime models.

import asyncio, sys
from livekit.agents import Agent, AgentSession, inference

async def run(model):
    is_realtime = model in ("google/gemini-3.8-live", "openai/gpt-realtime-2.1")
    llm = inference.RealtimeModel(model, modalities=["text"]) if is_realtime else inference.LLM(model)
    session = AgentSession(llm=llm)
    states = []
    session.on("agent_state_changed", lambda ev: states.append(ev.new_state))
    await session.start(agent=Agent(instructions="You are a friendly interviewer."))
    await session.generate_reply(instructions="Ask the user one short question.")
    await asyncio.sleep(2)
    print(f"=== {model}: states {states} -> final {session.agent_state!r}")
    await session.aclose()

for m in sys.argv[1:]:
    asyncio.run(run(m))

# python repro.py google/gemma-4-31b-it google/gemini-3.8-live openai/gpt-realtime-2.1

Output:

=== google/gemma-4-31b-it: states ['listening', 'thinking', 'speaking', 'listening'] -> final 'listening'
=== google/gemini-3.8-live: states ['listening', 'speaking'] -> final 'speaking'
=== openai/gpt-realtime-2.1: states ['listening', 'speaking'] -> final 'speaking'

The reply text is delivered in every case. Only the state is stuck, and it happens with both realtime providers.

Operating System

macOS 15

Models Used

LiveKit Inference: google/gemini-3.8-live, openai/gpt-realtime-2.1 (realtime, modalities=["text"]); google/gemma-4-31b-it as the pipeline comparison

Package Versions
livekit-agents==1.8.8
# also present on main (597e6b49)
Session/Room/Call IDs

No response

Proposed Solution
# In AgentActivity._realtime_generation_task_impl, update the agent state whether or not
# there's an audio output, and keep only the audio-specific work under the check:
self._session._update_agent_state("thinking" if self._background_speeches else "listening")
if audio_output is not None:
    self._on_end_of_agent_speech(ended_at=time.time())
    if self.interruption_enabled:
        self.interruption_by_audio_activity_enabled = True
    current_span.set_attribute(trace_types.ATTR_SPEECH_INTERRUPTED, speech_handle.interrupted)
Additional Context

Text-only realtime sessions are useful when the realtime model's text output is spoken by a separate TTS, or when the same agent serves both voice and text interviews. The only workaround we've found is setting the state to listening through the private _update_agent_state when a realtime reply finishes without an audio output.

Screenshots and Recordings

No response

Dominant language
Python
Stars
14.6k
Forks
3.9k
Avg merge
3d 1h
Merged PRs (30d)
173

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from livekit/agents

All issues in livekit/agents

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.