Enable streaming chat responses (SSE) with graceful fallback

Summary

Implement streaming chat responses so users see answers token-by-token with faster perceived latency. Use Server-Sent Events (SSE) from the backend, with an automatic fallback to the current non-streaming flow when streaming isn’t available. Include a “Stop generating” control, partial rendering of expert slices (optional phase 2), and basic telemetry (first-token time, total stream duration).

This feature aligns with our architecture (stateless API, MoE, RAG gating) and previously noted non-goal (“streaming could be enabled”)—this issue proposes delivering it safely behind a feature flag.

---

Motivation

• Reduce perceived latency (target: p50 first token < 700ms, p95 < 1.5s without RAG).
• Improve UX for long answers and property analyses.
• Maintain parity with modern AI chat experiences.
• Gather timing metrics to inform future optimizations (RAG topK, expert parallelism).

---

Scope

Backend (Express + TS)

• Add SSE support to chat endpoint, gated by query or header:

  • Endpoint: POST /api/chat?stream=true
  • Header: Accept: text/event-stream

• Stream events with types:

  • meta (request id, model, expert list, gate decision)
  • delta (partial text tokens)
  • expert_delta (optional phase 2: stream expert slices individually)
  • final (complete message, citations/visualization payloads)
  • error (structured)
  • metrics (latency, tokens)

• Respect JWT auth, CORS, and Redis rate limits (limit concurrent streams per user/IP).

• Timeouts: soft cancel at 30s server-side; heartbeat every 15s to keep connections alive.

• Add cancelation: client sends DELETE /api/chat/{requestId} or closes connection → propagate abort to providers.

• Prometheus: ai_stream_first_token_ms, ai_stream_duration_ms, ai_stream_active_gauges.

Frontend (Next.js + React)

• Add useChatStream hook using fetch ReadableStream (preferred) with fallback to EventSource.

• UI:

  • Live typing animation with monospace caret.
  • “Stop generating” button → cancels stream.
  • Auto-scroll with user override (pause on scroll).
  • Show partial expert pane badges; full per-expert streaming is phase 2.

• Settings toggle for streaming (user-level), default on when server flag enabled.

Feature Flag & Config

• STREAMING_ENABLED=true|false (backend)
• NEXT_PUBLIC_STREAMING_DEFAULT=true|false (frontend)
• Optional: STREAM_EXPERT_PANES=false (phase 2)

---

Non-Goals

• WebSockets (SSE only for now).
• Token-level visualization charts.
• Rewriting CI/CD—only minimal config/env additions.

---

Acceptance Criteria

• When STREAMING_ENABLED=true and client requests stream=true:

  • Responses render incrementally in the chat UI.
  • “Stop generating” cancels within ≤ 500ms and no orphan processes remain.
  • If SSE fails mid-flight, client falls back to buffered final response with a small banner.

• Metrics exposed:

  • ai_stream_first_token_ms and ai_stream_duration_ms recorded for ≥ 95% of requests.

• Security/limits:

  • Max concurrent streams per user enforced (default 2).
  • Idle connections close after heartbeat grace period.

• No regression to non-streaming users/clients.

---

API Contract (Proposed)

Request

POST /api/chat?stream=true
Authorization: Bearer <JWT>
Accept: text/event-stream
Content-Type: application/json

{
  "conversationId": "abc123",
  "message": "Find 3BR near Southern Village under $700k",
  "options": { "expertMode": true }
}

SSE Frames (examples)

event: meta
data: {"requestId":"rq_123","experts":["data","finance"],"rag":true}

event: delta
data: {"text":"Here are a few options near "}

event: delta
data: {"text":"Southern Village that match your budget..."}

event: final
data: {"messageId":"m_456","content":"...full text...", "experts":[...], "citations":[]}

event: metrics
data: {"firstTokenMs":540,"durationMs":2840}

event: error
data: {"code":"PROVIDER_TIMEOUT","message":"Gemini timed out, partial output returned"}

---

Telemetry & Observability

• New counters/histograms:

  • ai_stream_first_token_ms
  • ai_stream_duration_ms
  • ai_stream_events_total{type}
  • ai_stream_canceled_total

• Add x-request-id to all stream frames (echo in logs).

---

Risks & Mitigations

• Proxy buffering (ALB/CDN) → ensure Cache-Control: no-transform, Content-Type: text/event-stream, disable response buffering where applicable.
• Long-lived connections → enforce max duration, heartbeats, backpressure.
• Provider variability → if provider doesn’t stream, chunk local buffer every 50–100 tokens.

---

Dependencies

• None hard. Optional: lightweight SSE library for Node; otherwise native res.flush().

---

Testing Plan

• Unit: SSE controller emits frames in correct order; cancelation propagates.
• Integration: end-to-end stream including RAG on/off, expert count variations.
• E2E (Cypress): UI renders partial text; cancel stops generation; fallback works when network is cut.
• Load: 200 concurrent streams sustain without memory leak; p95 first-token & duration recorded.

---

Rollout Plan

1. Ship backend behind STREAMING_ENABLED=false.
2. Dark-launch on staging; validate metrics/dashboards.
3. Enable for 10% traffic; monitor errors and p95.
4. Ramp to 100% if stable.

---

Definition of Done

• Flags present and default-safe.
• Frontend and backend support incremental streaming with cancelation.
• Dashboards show first-token and duration metrics with meaningful data.
• Documentation added to README.md (Usage) and TECH_DOCS.md (API details).
• No regressions in non-streaming path; CI green.

---

Tasks

• Backend: SSE controller + headers + heartbeat
• Backend: Provider adapter with token callbacks
• Backend: Cancelation & timeouts
• Backend: Prometheus metrics
• Frontend: useChatStream hook
• Frontend: Chat UI incremental renderer + cancel button
• Frontend: Settings toggle & persisted preference
• Docs: README/TECH_DOCS updates
• Tests: unit, integration, E2E, load smoke
• Dashboards: first-token and duration panels

Estimate: 3–5 dev-days (split FE/BE), plus 1 day for polish & rollout.