Anthropic plugin: Claude Sonnet 5 and Haiku 5.5 return no text on default settings

Start in livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anthropic/llm.py: the default max_tokens (about line 183) and the request timeout passed from conn_options in chat() (lines 262-278). Run the reproduction script to see empty replies and timeouts. Done means the defaults are chosen per model, unlisted models keep the API default, and the script shows text back on Sonnet 5, Haiku 5.5, Sonnet 5.5 and Opus 5.5. Agree the per-model approach with maintainers first, since the options conflict.

Bug Description

Claude Sonnet 5, Sonnet 5.5, Haiku 5.5 and Opus 5.5 think by default when a request has no thinking field. The Anthropic plugin sends no thinking setting and caps each response at 1024 tokens unless max_tokens is set, so on default settings these models think inside that cap. On a question that needs some working out:

• No reply. Sonnet 5 and Haiku 5.5 often spend the whole 1024 tokens thinking and stop with stop_reason: "max_tokens" before writing any text, so the agent says nothing.
• Timeouts. chat() passes conn_options.timeout (10 s by default) to the SDK as the request timeout (here). Sonnet 5 can think for longer than that without sending text, so each attempt times out, and after the retries the plugin raises APIConnectionError: failed to generate LLM completion after 4 attempts (caused by TimeoutError).
• Long pauses. Sonnet 5.5 and Opus 5.5 do answer, but about 5 seconds after the question.

Through the plugin on main (4814a8f), the script below, 4 requests per model:

claude-sonnet-5: no text after 21 s
claude-sonnet-5: APIConnectionError after 47 s: failed to generate LLM completion after 4 attempts (caused by TimeoutError)
claude-sonnet-5: APIConnectionError after 47 s: failed to generate LLM completion after 4 attempts (caused by TimeoutError)
claude-sonnet-5: no text after 11 s
claude-haiku-5-5: no text after 5 s
claude-haiku-5-5: no text after 5 s
claude-haiku-5-5: no text after 5 s
claude-haiku-5-5: no text after 5 s

With conn_options=APIConnectOptions(timeout=90, max_retry=0) the Sonnet 5 timeouts go away, but the replies still come back empty after about 15 s: the thinking fills the 1024-token cap either way.

The same request on the API directly with max_tokens=1024 came back with no text in 5 of 5 runs on Sonnet 5 and 4 of 5 on Haiku 5.5.

Expected Behavior

On default settings, each of these models answers the question, and starts within about a second as it does with thinking off.

Reproduction Steps

1. Set ANTHROPIC_API_KEY.
2. Run the script below.
3. Most requests end with no text or with a timeout error.

import asyncio
import time

from livekit.agents import llm
from livekit.plugins import anthropic

PROMPT = (
    "I need a 45-minute call between a colleague in Helsinki and one in San Francisco next "
    "Tuesday. Neither wants to meet before 8am or after 6pm their time. When should it start?"
)


async def reply(model: str) -> str:
    chat_ctx = llm.ChatContext()
    chat_ctx.add_message(role="user", content=PROMPT)
    start = time.monotonic()
    first, text = None, ""
    try:
        async with anthropic.LLM(model=model).chat(chat_ctx=chat_ctx) as stream:
            async for chunk in stream:
                if chunk.delta and chunk.delta.content:
                    first = first or time.monotonic() - start
                    text += chunk.delta.content
    except Exception as e:
        return f"{model}: {type(e).__name__} after {time.monotonic() - start:.0f} s: {e}"
    if not text.strip():
        return f"{model}: no text after {time.monotonic() - start:.0f} s"
    return f"{model}: first text after {first:.1f} s, {len(text)} characters"


async def main():
    runs = [reply(model) for model in ("claude-sonnet-5", "claude-haiku-5-5") for _ in range(4)]
    for line in await asyncio.gather(*runs):
        print(line)


asyncio.run(main())

Operating System

macOS 27.0.1

Models Used

claude-sonnet-5, claude-haiku-5-5 (and claude-sonnet-5-5, claude-opus-5-5 for the latency)

Package Versions

livekit-agents and livekit-plugins-anthropic from main at 4814a8f (run from source)
anthropic==1.13.0
python 3.13.16

Proposed Solution

I measured the settings that could serve as defaults, on the API directly, same question, 5 runs each. Each cell is replies with no text, then the median time to first text:

Setting	Sonnet 5	Haiku 5.5	Sonnet 5.5	Opus 5.5
Defaults, max_tokens=1024 (today)	5/5 empty	4/5 empty	0/5, 5.5 s	0/5, 5.5 s
max_tokens=4096	2/5 empty, 43 s	0/5, 6.9 s
effort: "low"	2/5 empty, 4.5 s	5/5 empty	0/5, 0.5 s	0/5, 3.8 s
thinking: {"type": "disabled"}	0/5, 0.55 s	0/5, 0.51 s	400	400

No single setting works for all four: disabled fixes Sonnet 5 and Haiku 5.5 but Sonnet 5.5 and Opus 5.5 reject it, and effort: "low" fixes Sonnet 5.5 but not Haiku 5.5. Raising max_tokens alone removes the empty replies on Haiku 5.5 but not the wait.

So the defaults probably need to be chosen per model. The open pipecat-ai/pipecat#6108 takes that approach for Pipecat, with a table keyed by exact model name, so an unlisted or future model gets the API's default rather than a value it may reject. Under that approach the table above suggests disabled for Sonnet 5 and Haiku 5.5, which removes the empty replies and the wait. Sonnet 5.5 and Opus 5.5 answered on defaults, just slowly, so I'd leave them at the API default.

Edit (Oct 12): I originally also suggested effort: "low" for Sonnet 5.5, based on latency only. Checking answers since then, I no longer would: at low effort Sonnet 5.5 usually opened with the wrong yes/no on reasoning questions and corrected itself mid-reply. disabled on Sonnet 5 and Haiku 5.5 keeps tool calls as accurate and faster, but has a milder version of the same problem on reasoning turns without a tool (data in pipecat-ai/pipecat#6108 (comment)).

Additional Context

• #6868 also changes these defaults, but sends thinking: {"type": "disabled"} to every model from 4.6 up, which Sonnet 5.5 and Opus 5.5 reject with a 400, as @iRajul noted there for Sonnet 5.5.
• #5529 set the client's read timeout to 30 s for thinking models, but chat() passes the 10 s conn_options.timeout per request, so that read timeout doesn't apply to it. With the default, Sonnet 5's attempts failed after about 14 s each.
• Anthropic's docs say max_tokens "covers thinking plus text" (Sonnet 5.5 migration guide).
• The plugin ran from source with stand-ins for packages unrelated to the LLM call. I'm happy to test a fix.