Improve intent recognition and disambiguation for similarly phrased user queries

The chatbot frequently misclassifies user intent when queries are semantically close or phrased in a superficially similar way, leading to incorrect responses, unnecessary clarification loops, and degraded user experience. Examples include:

• “What is the rent?” vs. “How do I pay rent?”
• “Tell me about the apartments.” vs. “Schedule a tour of the apartments.”

Such confusions indicate gaps in the current NLU pipeline’s ability to (1) represent nuanced differences between intents, (2) gauge uncertainty, and (3) resolve ambiguity dynamically in conversation.

Goals:

1. Accurately distinguish between borderline/overlapping intents in user queries.
2. Surface and resolve ambiguity proactively when the model is uncertain.
3. Prevent regressions via automated testing and continuous feedback.
4. Establish observability so intent classification failures are detectable, diagnosable, and looped back into improvement.

Scope (In-Scope):

• Auditing and augmenting training data for confusing intent pairs.
• Upgrading or enhancing the intent classification model (e.g., transformer-based embeddings).
• Implementing confidence estimation and dynamic clarification / slot-filling when intent ambiguity is detected.
• Building automated test coverage for ambiguous queries and regression prevention.
• Instrumenting telemetry to monitor real-world intent errors and incorporating explicit/implicit feedback.
• Updating docs and guidelines for intent labeling and disambiguation strategies.

Out of Scope:

• Full rewrite of dialogue management unrelated to intent disambiguation (unless needed as part of clarification flow).
• One-off manual relabeling of all historical logs (should be handled via a prioritized backlog or separate cleanup initiative).

Proposed Solution Areas:

1. Data Audit & Expansion:

   • Analyze existing intent labels to identify overlapping/confusable intent pairs.
   • Generate and curate additional labeled examples (including paraphrases, user rephrases, and edge cases) for these pairs.
   • Establish guidelines for new intent examples to reduce future overlap.

2. Model & Representation Improvements:

   • Evaluate current classifier baseline.
   • Prototype transformer-based embedding approaches (e.g., fine-tuned sentence transformers or intent-specialized BERT variants) to better capture semantic nuance.
   • Calibrate confidence scores (temperature scaling, Bayesian approximations, etc.) to make uncertainty meaningful.
   • Compare performance against baseline with quantitative metrics.

3. Dynamic Disambiguation / Clarification:

   • Define confidence thresholds that trigger a lightweight clarification or slot-filling dialog instead of a potentially wrong hard classification.
   • Design conversational fallback patterns (e.g., “Did you mean X or Y?”, “I can tell you about the apartments or help schedule a tour— which would you like?”).
   • Support multi-intent detection when appropriate and allow users to refine.

4. Testing & Validation:

   • Create unit and integration tests targeting known ambiguous query pairs.
   • Automate generation of borderline cases for regression suites.
   • Include evaluation harness to surface confusion matrices and per-intent precision/recall as part of CI.

5. Observability & Feedback Loop:

   • Instrument implicit signals (rephrases, low engagement, correction phrases) and explicit feedback for misclassification detection.
   • Surface frequent failure modes in dashboards (top misclassified intent pairs, trends, spike alerts).
   • Human-in-the-loop vetting pipeline to validate high-value feedback and promote it into training data.

6. Documentation & Process:

   • Update developer documentation about intent design, ambiguity handling, clarification flow, and retraining procedures.
   • Define a cadence for reviewing confusion patterns and retraining with validated corrections.

Acceptance Criteria:

• Ambiguous intent pairs are identified and enriched with high-quality labeled data.
• A new or enhanced intent classifier (with improved embedding/representation) demonstrates measurable reduction in misclassification on validation sets, especially for previously overlapping intents.
• Confidence-aware clarification flow is implemented and triggered appropriately when uncertainty is above threshold.
• Automated tests for borderline queries exist, are part of CI, and catch regressions.
• Telemetry surfaces intent errors, with a working dashboard, alerting for spikes, and a process for ingesting validated feedback into training.
• Documentation reflects the new data, model, and operational practices.

Success Metrics:

• % reduction in misclassification rate for top N ambiguous intent pairs (baseline → new).
• Improved intent classification F1-score overall and on edge-case subsets.
• Decrease in user rephrases / correction follow-ups after initial response.
• Volume and quality of feedback incorporated (e.g., number of validated corrections added to training).
• Time-to-detect spikes in intent confusion (alert responsiveness).
• User satisfaction metrics tied to intent understanding (if available, e.g., “Was this helpful?” rates for intent-sensitive flows).

Risks & Mitigations:

• Risk: Transformer models increase latency. Mitigation: Benchmark and consider distillation / caching / hybrid designs with fallback to lightweight model.
• Risk: Over-clarifying annoys users. Mitigation: Tune thresholds; make clarification succinct and context-aware; fall back to best guess if user signals impatience.
• Risk: Feedback noise polluting training data. Mitigation: Human validation pipeline and confidence weighting.

Dependencies:

• Access to historical conversation logs and intent labeling tooling.
• Infrastructure for model training/evaluation.
• Dialog manager support for clarification flows.
• Analytics/dashboarding platform for telemetry.

Related Subissues:

• Enhance intent disambiguation for similarly phrased queries (model/data/clarification).
• Build user feedback loop and analytics for intent classification errors.

Rollout Plan:

• Phase 1: Data audit + prototype improved intent classifier; offline evaluation.
• Phase 2: Implement confidence-aware clarification flow; integrate into a canary subset of traffic.
• Phase 3: Full deployment with monitoring, automated test guardrails, and periodic retraining cycle based on feedback.

---