Feature: Evaluation and Experimentation Platform (Offline Benchmarks + Online Feedback Loops)

Summary

Build a repeatable evaluation/experimentation layer to compare prompt templates, retrieval strategies, and model/provider settings using offline datasets and online user feedback.

Why this matters

Without disciplined experiments, feature work can regress answer quality or latency. The project needs a systemized way to decide what should ship.

Scope

• Versioned experiment configs (prompt, retriever mode, reranker, model, top-k).
• Offline benchmark runner with quality + latency/cost metrics.
• Golden dataset management and regression thresholds.
• Online feedback capture in UI (helpful, not helpful, optional reason).
• Decision dashboard/report artifact for release gates.

Non-goals

• Full enterprise analytics warehouse integration.
• Fully automated self-tuning in v1.

Proposed implementation

1. Define experiment manifest format and runner CLI.
2. Create benchmark dataset schema + seed set for portfolio QA tasks.
3. Compute metrics (faithfulness, relevance, source coverage, latency, token cost).
4. Add frontend feedback controls and backend storage.
5. Merge offline + online signals into a release recommendation report.

Acceptance criteria

• Experiments are reproducible from a checked-in manifest.
• CI can run a smoke benchmark and fail on configured regression thresholds.
• UI captures user feedback linked to response/request IDs.
• Weekly report artifact summarizes top configurations and tradeoffs.
• Documentation describes how to run experiments and interpret outcomes.

Relationship to existing issues

• Extends issue #3’s minimal eval direction into a full experimentation framework with release decision support.

Labels

enhancement, evaluation, mlops, frontend, backend