Feature: Stabilize ML Inference Under Low Memory/CPU (Queues, Limits, Caching, Circuit Breakers)

Summary

Moodify’s free-tier backend (0.1 CPU / 512 MB) occasionally restarts or stalls during speech/facial inference and bursts of requests. Let’s harden the inference path with a small set of resource-aware features: a Redis-backed job queue, request limits/timeouts, output caching, and circuit breakers—plus a graceful UI/UX for fallbacks.

Goals

• Prevent OOM/restarts during ML jobs.
• Keep the API responsive under load with graceful degradation (fallback to text model).
• Provide users progress/clear errors when heavy jobs queue or time out.

Non-Goals

• Changing model architectures or re-training. (Future work possible.)

---

Scope & Design

1) Move heavy inference to a queued worker

• Add Redis + RQ/Celery workers for speech_emotion, facial_emotion.

  • Location: ai_ml/src/api/ and backend/api/ glue.
  • New queue: inference:default.
  • Jobs accept payload (text/audio/image), return {emotion, confidence, meta}.

• Synchronous facade: API returns a jobId immediately for heavy tasks; client polls /api/jobs/:id or opens a WS channel for progress.

2) Input caps & preprocessing

• Server-side limits (Django settings + Nginx):

  • Audio max 20 MB / 60 s; Image max 5 MB / 4096×4096.

• Auto downsample audio to 16 kHz mono; resize images to model’s target (e.g., 224×224) before enqueue.

• Reject with 413 Payload Too Large + actionable message.

3) Timeouts, retries, circuit breakers

• Per-job timeout (e.g., 20–30 s speech, 10–15 s facial).
• Retry policy: 1 immediate + 1 delayed retry (jitter).
• Circuit breaker on each route: open after N failures / high queue depth; while open, return 503 + suggest text-based analysis.

4) Response caching (short-lived)

• Cache key: SHA256 of normalized input (and model version).
• Store in Redis for 15–60 minutes to dedupe repeats and speed up replays.

5) Backpressure & graceful fallback

• If queue length > threshold (e.g., 50 jobs), API responds 429 with Retry-After and offers fallback to text model via a link/flag the frontend can honor.

6) Observability

• Prometheus counters/gauges: inference_queue_depth, inference_duration_seconds, inference_timeouts_total, inference_circuit_open_total, inference_cache_hits_total.
• Structured logs: include jobId, route, elapsedMs, cached, timeout, retries.
• Basic health endpoint checks Redis + worker heartbeat.

7) Frontend UX (minimal)

• When user submits speech/facial: show “Queued… [position X]”, then “Processing…”, then result.
• If breaker open or 429, show “Server busy—try text emotion instead” with one-click switch.

---

Acceptance Criteria

• ✅ Under a 100-request burst (speech/facial), API stays responsive; no container restarts.
• ✅ Jobs exceeding timeout are marked failed; users see a clear message; the system does not hang.
• ✅ Identical inputs within cache TTL return instantly (cache hit metric increments).
• ✅ Circuit opens after repeated timeouts or long queues; closes automatically after cool-down.
• ✅ Metrics exposed at /metrics; basic Grafana dashboard JSON checked in.
• ✅ Frontend shows progress/fallback paths; no spinner-forever states.

---

Tasks

Backend / Infra

• Add Redis service (docker-compose; k8s optional).
• Implement RQ/Celery workers in ai_ml/src/api/worker.py (or similar).
• Wrap Django endpoints (/api/speech_emotion/, /api/facial_emotion/) to enqueue & return jobId.
• Add /api/jobs/:id (GET) and optional WS channel for job status/progress.
• Preprocessing: audio downsample (ffmpeg/librosa), image resize (Pillow/OpenCV).
• Input size guards; return 413/422 with hints.
• Add timeout/retry policy at worker layer.
• Implement circuit breaker (simple Redis counters + sliding window).
• Add Redis caching of outputs keyed by input hash + model version.
• Expose Prometheus metrics; wire basic alerts (timeouts spike, queue depth).

Frontend

• Job status polling + progress UI.
• Fallback CTA to text analysis on 429/503 or breaker open.
• Friendly errors for 413/422 with “How to fix” tips.

DevOps

• Extend docker-compose.yml with Redis + worker service.
• CI step to run worker unit tests.
• Optional k8s manifests for worker Deployment + HPA (CPU-based).
• Basic Grafana dashboard JSON in data_analytics/visualizations or docs/.

Tests

• Unit: input caps, cache keying, breaker open/close logic.
• Integration: enqueue → completes; enqueue → timeout → retry → DLQ/failed.
• Load: 100 concurrent speech jobs; ensure no OOM/restarts; track p95 latency of API facade.
• E2E: frontend shows queue position → result; fallback path works.

---

Risks & Mitigations

• Redis unavailable → short-circuit routes to text-only with 503 and log an alert.
• Cache poisoning → include model version + normalized preprocessing parameters in hash.
• User uploads very long audio → strict server-side duration cut + early 413.

---