Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

infra: agent backend cannot scale beyond a few concurrent users

Abierto
  • ¿Algo no está bien?
#65 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Tranquilo
Stack tecnológico
postgresql, python, redis, sqlite, typescript
Área
backend, cloud, database, devops, frontend, infrastructure

Línea de trabajo

Start with apps/agent/main.py, apps/agent/src/bounded_memory_saver.py, render.yaml, and apps/app/src/app/api/copilotkit/route.ts; reproduce the concurrency failure described in #63 before choosing a phase. The work is done only when the selected scaling, persistence, backpressure, or load-testing goals are implemented and verified at the stated target concurrency.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

The agent backend runs on a single uvicorn worker process with an in-memory checkpointer on a 512MB Render starter instance. This is a global bottleneck — not per-user. All concurrent users share the same event loop, the same memory pool, and the same 200-thread checkpoint limit.

Currently the app breaks at ~3 concurrent connections (#63). At production scale (100-1000 users), it would be effectively unusable.

Architecture bottlenecks

1. Single worker process
  • uvicorn runs with 1 worker (default) — all requests share one Python event loop
  • Each GPT-5.4 visualization call takes 10-30s
  • LangGraph has synchronous sections that block the event loop
  • Throughput: ~2-6 visualization requests/minute
2. In-memory checkpointer (BoundedMemorySaver)
  • All conversation state stored in RAM — shared global pool of 200 threads
  • FIFO eviction: after 200 conversations across ALL users, oldest threads are silently deleted
  • Users lose conversation context mid-session with no error
  • Not thread-safe — designed for single-process async only
  • On 512MB starter plan, memory pressure builds well before 200 threads
3. No backpressure or error surfacing
  • When the backend is saturated, requests hang silently — no timeout, no error, no retry
  • Frontend shows no indication that the agent is overloaded
  • Health check at /health returns 200 even when the event loop is blocked

Scale projections

Concurrent users Behavior
1-5 Works fine
10-20 Noticeable latency, requests queue
50+ Requests timeout, SSE connections drop
100+ Effectively down, health checks fail, Render restarts

Proposed solution

Phase 1 — Quick wins (config changes only)
  • Add --workers 4 to uvicorn startCommand in render.yaml — multiplies throughput ~4x
  • Upgrade agent service from starter (512MB) to standard (1GB+) in render.yaml
  • Enable rate limiting (RATE_LIMIT_ENABLED=true) with reasonable limits (e.g. 20 req/min per IP)
Phase 2 — Persistent checkpointer
  • Replace BoundedMemorySaver with PostgreSQL or SQLite async checkpointer
  • Conversation state survives restarts and doesn't consume RAM
  • No more silent thread eviction — threads persist until explicitly cleaned up
  • Render already supports managed Postgres — can add as a service in render.yaml
Phase 3 — Error handling and backpressure
  • Add frontend timeout — show error after ~30s of no response instead of hanging forever
  • Add backend concurrency limit — return 503 "busy" when at capacity rather than queuing indefinitely
  • Add connection health monitoring — detect dropped SSE connections and surface to user
  • Reuse thread IDs per browser tab (sessionStorage) to avoid creating unnecessary threads
Phase 4 — Horizontal scaling
  • Use Gunicorn with uvicorn workers for proper process management
  • Verify Render auto-scaling (1-3 instances) works correctly with persistent checkpointer
  • Add Redis or Postgres for shared state across instances
  • Load test at target concurrency (100+ users) to validate

Related issues

  • #63 — Agent stops responding after multiple concurrent tabs (symptom of this)
  • #58 — Quality regression (long-running visualization calls exacerbate the single-worker bottleneck)
  • #62 — Planning step before visualization (adds an extra round-trip, making concurrency even more critical)

Key files

  • apps/agent/main.py — uvicorn config, BoundedMemorySaver(max_threads=200)
  • apps/agent/src/bounded_memory_saver.py — FIFO eviction logic
  • render.yaml — Render service config (starter plan, no worker config)
  • apps/app/src/app/api/copilotkit/route.ts — Frontend → agent connection
Lenguaje dominante
TypeScript
Estrellas
3.9k
Forks
414
Merge medio
4 h
PR fusionados (30 d)
6

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de CopilotKit/OpenIntelligentUI

Todos los issues de CopilotKit/OpenIntelligentUI

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.