Citation Quality Analytics and User Feedback Loop for AI Responses

Summary

Add a dedicated analytics and feedback system for citation quality in AI-generated responses, so teams can measure reliability and continuously improve grounding behavior.

Problem

The chatbot already emits citations and guardrail warnings, but there is no operational dashboard to quantify citation quality over time or collect structured user feedback on citation correctness. This makes it difficult to prioritize prompt/model/retrieval improvements with evidence.

Proposed Solution

• Track citation KPIs per response and aggregate by day/week/source/model.
• Build an internal analytics dashboard focused on citation integrity and guardrail outcomes.
• Add in-product feedback controls for users to flag incorrect or weak citations.
• Route flagged responses to a review queue with triage status and resolution notes.
• Provide exportable reports for reliability reviews.

Acceptance Criteria

• Backend stores per-response metrics such as citationCount, invalidCitationCount, missingCitationWarning, numericClaimWarning, and confidenceWithoutCitationWarning.
• Dashboard includes trend charts and filters (date range, source, route, model version).
• Users can submit feedback type (incorrect, insufficient, irrelevant) for a specific AI response.
• Review workflow supports states: new, triaged, resolved, dismissed.
• Monthly CSV export is available for governance reporting.

Technical Notes

• Backend:
  • Add citation_metrics and citation_feedback collections.
  • Add endpoints:
    • GET /api/admin/citations/metrics
    • GET /api/admin/citations/feedback
    • POST /api/citations/feedback
    • PATCH /api/admin/citations/feedback/:id
• Frontend:
  • Admin page: citation dashboard + feedback queue.
  • Chat UI: small "Report citation issue" action per AI message.
• Observability:
  • Emit structured logs/events for guardrail outcomes.

Tasks

• Define metric schema and aggregation jobs.
• Implement feedback submission and moderation endpoints.
• Build citation analytics dashboard with filters and trend views.
• Add user feedback action in chat message UI.
• Build feedback triage interface for admins/reviewers.
• Add monthly CSV export endpoint and access control.
• Add tests for metric ingestion, aggregation, and triage workflow.
• Document KPI definitions and interpretation guide.

Out of Scope (Initial Iteration)

• Auto-retraining models from feedback.
• Third-party BI tool integration.