Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

infra: agent backend cannot scale beyond a few concurrent users

Aberta
  • Algo errado?
#65 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
30/100
Tipo de issue
Funcionalidade
Clareza
Precisa de esclarecimento
Status de atividade
Pouca atividade
Stack de tecnologia
postgresql, python, redis, sqlite, typescript
Domínio
backend, cloud, database, devops, frontend, infrastructure

Direção de pesquisa

Start with apps/agent/main.py, apps/agent/src/bounded_memory_saver.py, render.yaml, and apps/app/src/app/api/copilotkit/route.ts; reproduce the concurrency failure described in #63 before choosing a phase. The work is done only when the selected scaling, persistence, backpressure, or load-testing goals are implemented and verified at the stated target concurrency.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Problem

The agent backend runs on a single uvicorn worker process with an in-memory checkpointer on a 512MB Render starter instance. This is a global bottleneck — not per-user. All concurrent users share the same event loop, the same memory pool, and the same 200-thread checkpoint limit.

Currently the app breaks at ~3 concurrent connections (#63). At production scale (100-1000 users), it would be effectively unusable.

Architecture bottlenecks

1. Single worker process
  • uvicorn runs with 1 worker (default) — all requests share one Python event loop
  • Each GPT-5.4 visualization call takes 10-30s
  • LangGraph has synchronous sections that block the event loop
  • Throughput: ~2-6 visualization requests/minute
2. In-memory checkpointer (BoundedMemorySaver)
  • All conversation state stored in RAM — shared global pool of 200 threads
  • FIFO eviction: after 200 conversations across ALL users, oldest threads are silently deleted
  • Users lose conversation context mid-session with no error
  • Not thread-safe — designed for single-process async only
  • On 512MB starter plan, memory pressure builds well before 200 threads
3. No backpressure or error surfacing
  • When the backend is saturated, requests hang silently — no timeout, no error, no retry
  • Frontend shows no indication that the agent is overloaded
  • Health check at /health returns 200 even when the event loop is blocked

Scale projections

Concurrent users Behavior
1-5 Works fine
10-20 Noticeable latency, requests queue
50+ Requests timeout, SSE connections drop
100+ Effectively down, health checks fail, Render restarts

Proposed solution

Phase 1 — Quick wins (config changes only)
  • Add --workers 4 to uvicorn startCommand in render.yaml — multiplies throughput ~4x
  • Upgrade agent service from starter (512MB) to standard (1GB+) in render.yaml
  • Enable rate limiting (RATE_LIMIT_ENABLED=true) with reasonable limits (e.g. 20 req/min per IP)
Phase 2 — Persistent checkpointer
  • Replace BoundedMemorySaver with PostgreSQL or SQLite async checkpointer
  • Conversation state survives restarts and doesn't consume RAM
  • No more silent thread eviction — threads persist until explicitly cleaned up
  • Render already supports managed Postgres — can add as a service in render.yaml
Phase 3 — Error handling and backpressure
  • Add frontend timeout — show error after ~30s of no response instead of hanging forever
  • Add backend concurrency limit — return 503 "busy" when at capacity rather than queuing indefinitely
  • Add connection health monitoring — detect dropped SSE connections and surface to user
  • Reuse thread IDs per browser tab (sessionStorage) to avoid creating unnecessary threads
Phase 4 — Horizontal scaling
  • Use Gunicorn with uvicorn workers for proper process management
  • Verify Render auto-scaling (1-3 instances) works correctly with persistent checkpointer
  • Add Redis or Postgres for shared state across instances
  • Load test at target concurrency (100+ users) to validate

Related issues

  • #63 — Agent stops responding after multiple concurrent tabs (symptom of this)
  • #58 — Quality regression (long-running visualization calls exacerbate the single-worker bottleneck)
  • #62 — Planning step before visualization (adds an extra round-trip, making concurrency even more critical)

Key files

  • apps/agent/main.py — uvicorn config, BoundedMemorySaver(max_threads=200)
  • apps/agent/src/bounded_memory_saver.py — FIFO eviction logic
  • render.yaml — Render service config (starter plan, no worker config)
  • apps/app/src/app/api/copilotkit/route.ts — Frontend → agent connection
Linguagem predominante
TypeScript
Estrelas
3.9k
Forks
414
Merge médio
4h
PRs com merge (30d)
6

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de CopilotKit/OpenIntelligentUI

Todas as issues de CopilotKit/OpenIntelligentUI

Issues semelhantes

Mais issues de TypeScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.