aws: LLM opens a new bedrock-runtime client every turn, so each reply waits on a fresh TCP+TLS handshake

Abierto
  • ¿Algo no está bien?
#7,419 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@amishaguptaberk ya está trabajando en esto.

Desde el 13/10/2026.

  • #7437 de @lzcGeek — cerrado sin fusionar
  • #7772 de @amishaguptaberk — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
aws, python
Área
backend-api-design, performance

Línea de trabajo

Start in livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/llm.py, especially LLMStream._run at line 324, and compare how the OpenAI and Google plugins retain and close clients. Work out client ownership and how prewarming can open a Bedrock connection; the issue leaves those design choices unresolved. Done means one client is reused across turns, closed in aclose(), and prewarming is supported.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Bug Description

Thanks for the LLM.prewarm() hook from #6484. The anthropic, openai, google and mistralai plugins implement it; the AWS plugin can't yet, because it has no client that lives past one turn.

LLMStream._run enters self._session.create_client("bedrock-runtime", ...) on every chat() call (llm.py L324). So each turn builds a new client and a new TCP + TLS connection, then closes both. Timing OpenAI GPT-6 on Bedrock:

"Say pong.", us-east-1 new client per turn (today) one reused client
aws.LLM time to first token, us.openai.gpt-6-sol, n=10 median 1.10 s 0.69 s
aws.LLM time to first token, us.openai.gpt-6-luna, n=10 median 1.13 s 0.67 s
aiobotocore converse_stream end to end, Sol, 3 runs (n=5, 5, 10) 1.10 / 0.95 / 1.02 s 0.67 / 0.70 / 0.71 s

The "reused" aws.LLM rows monkeypatch create_client to hand back one already-open client. Split (n=10 medians): TCP connect 194 ms, TLS 1.3 handshake 199 ms, building the client 6 ms. So every turn pays two extra round trips: ~0.4 s from my machine (~190 ms from us-east-1), far less for an agent in the same region.

Expected Behavior

One bedrock-runtime client per aws.LLM instance, reused across turns and closed in aclose(), the way the openai and google plugins keep theirs. _prewarm_impl could then open the connection before the first reply.

Reproduction Steps
# aiobotocore directly, same Config the plugin uses; AWS credentials in the environment
uv run python - <<'PY'
import asyncio, statistics, time
from aiobotocore.session import get_session
from botocore.config import Config

REQ = dict(modelId="us.openai.gpt-6-sol",
           messages=[{"role": "user", "content": [{"text": "Say pong."}]}],
           inferenceConfig={"maxTokens": 64})
CFG = Config(user_agent_extra="x-client-framework:livekit-plugins-aws")

async def one(client):
    async for _ in (await client.converse_stream(**REQ))["stream"]:
        pass

async def main():
    s = get_session(); s.set_config_variable("region", "us-east-1")
    new, reuse = [], []
    async with s.create_client("bedrock-runtime", config=CFG) as shared:
        await one(shared)
        for _ in range(5):
            t0 = time.perf_counter()
            async with s.create_client("bedrock-runtime", config=CFG) as c:  # what _run does
                await one(c)
            new.append(time.perf_counter() - t0)
            t0 = time.perf_counter(); await one(shared); reuse.append(time.perf_counter() - t0)
    print(f"new client {statistics.median(new):.2f}s vs reused {statistics.median(reuse):.2f}s")

asyncio.run(main())
PY

new client 1.15s vs reused 0.72s
Operating System

macOS 26 (network-bound, not OS-specific)

Models Used

AWS Bedrock us.openai.gpt-6-sol via aws.LLM (any Bedrock model)

Package Versions
livekit-agents==1.8.2
livekit-plugins-aws==1.8.2   # main at b4df92e
aiobotocore==3.8.0
botocore==1.43.46
python==3.13
Session/Room/Call IDs

N/A

Proposed Solution

Create the client lazily on the first chat() (an AsyncExitStack on the LLM), reuse it in LLMStream._run, close it in aclose(), add _prewarm_impl. Two choices I'd leave to you: which event loop owns the client, and what a cheap request for _prewarm_impl would be (bedrock-runtime has no list-models call). Happy to open a PR once that's settled.

Additional Context
Related
  • Polly TTS has the same per-request pattern (tts.py L162). I did not measure it.
  • 250-word answers, n=3 medians: aws.LLM TTFT was 1.38 s (Sol) and 1.80 s (Luna). A reused boto3 client got 0.76 s and 1.32 s.
Lenguaje dominante
Python
Estrellas
14.6k
Forks
3.9k
Merge medio
2 d 7 h
PR fusionados (30 d)
224

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de livekit/agents

Todos los issues de livekit/agents

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.