Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

infra: agent backend cannot scale beyond a few concurrent users

Ouverte
  • Quelque chose ne va pas ?
#65 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
30/100
Type d'issue
Fonctionnalité
Clarté
À clarifier
Activité
Calme
Stack technique
postgresql, python, redis, sqlite, typescript
Domaine
backend, cloud, database, devops, frontend, infrastructure

Piste de recherche

Start with apps/agent/main.py, apps/agent/src/bounded_memory_saver.py, render.yaml, and apps/app/src/app/api/copilotkit/route.ts; reproduce the concurrency failure described in #63 before choosing a phase. The work is done only when the selected scaling, persistence, backpressure, or load-testing goals are implemented and verified at the stated target concurrency.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Problem

The agent backend runs on a single uvicorn worker process with an in-memory checkpointer on a 512MB Render starter instance. This is a global bottleneck — not per-user. All concurrent users share the same event loop, the same memory pool, and the same 200-thread checkpoint limit.

Currently the app breaks at ~3 concurrent connections (#63). At production scale (100-1000 users), it would be effectively unusable.

Architecture bottlenecks

1. Single worker process
  • uvicorn runs with 1 worker (default) — all requests share one Python event loop
  • Each GPT-5.4 visualization call takes 10-30s
  • LangGraph has synchronous sections that block the event loop
  • Throughput: ~2-6 visualization requests/minute
2. In-memory checkpointer (BoundedMemorySaver)
  • All conversation state stored in RAM — shared global pool of 200 threads
  • FIFO eviction: after 200 conversations across ALL users, oldest threads are silently deleted
  • Users lose conversation context mid-session with no error
  • Not thread-safe — designed for single-process async only
  • On 512MB starter plan, memory pressure builds well before 200 threads
3. No backpressure or error surfacing
  • When the backend is saturated, requests hang silently — no timeout, no error, no retry
  • Frontend shows no indication that the agent is overloaded
  • Health check at /health returns 200 even when the event loop is blocked

Scale projections

Concurrent users Behavior
1-5 Works fine
10-20 Noticeable latency, requests queue
50+ Requests timeout, SSE connections drop
100+ Effectively down, health checks fail, Render restarts

Proposed solution

Phase 1 — Quick wins (config changes only)
  • Add --workers 4 to uvicorn startCommand in render.yaml — multiplies throughput ~4x
  • Upgrade agent service from starter (512MB) to standard (1GB+) in render.yaml
  • Enable rate limiting (RATE_LIMIT_ENABLED=true) with reasonable limits (e.g. 20 req/min per IP)
Phase 2 — Persistent checkpointer
  • Replace BoundedMemorySaver with PostgreSQL or SQLite async checkpointer
  • Conversation state survives restarts and doesn't consume RAM
  • No more silent thread eviction — threads persist until explicitly cleaned up
  • Render already supports managed Postgres — can add as a service in render.yaml
Phase 3 — Error handling and backpressure
  • Add frontend timeout — show error after ~30s of no response instead of hanging forever
  • Add backend concurrency limit — return 503 "busy" when at capacity rather than queuing indefinitely
  • Add connection health monitoring — detect dropped SSE connections and surface to user
  • Reuse thread IDs per browser tab (sessionStorage) to avoid creating unnecessary threads
Phase 4 — Horizontal scaling
  • Use Gunicorn with uvicorn workers for proper process management
  • Verify Render auto-scaling (1-3 instances) works correctly with persistent checkpointer
  • Add Redis or Postgres for shared state across instances
  • Load test at target concurrency (100+ users) to validate

Related issues

  • #63 — Agent stops responding after multiple concurrent tabs (symptom of this)
  • #58 — Quality regression (long-running visualization calls exacerbate the single-worker bottleneck)
  • #62 — Planning step before visualization (adds an extra round-trip, making concurrency even more critical)

Key files

  • apps/agent/main.py — uvicorn config, BoundedMemorySaver(max_threads=200)
  • apps/agent/src/bounded_memory_saver.py — FIFO eviction logic
  • render.yaml — Render service config (starter plan, no worker config)
  • apps/app/src/app/api/copilotkit/route.ts — Frontend → agent connection
Langage dominant
TypeScript
Étoiles
3.9k
Forks
414
Merge moyen
4 h
PR mergées (30 j)
6

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de CopilotKit/OpenIntelligentUI

Toutes les issues de CopilotKit/OpenIntelligentUI

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.