Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Guardrails: Topic relevance evaluation

Open
  • Is something wrong?
#102 1 comment 0 reactions 1 assignee View on GitHub

@AkhileshNegi is already working on this.

Since Apr 30, 2026.

Assessment

This issue has not been assessed yet.

Description

Is your feature request related to a problem?
We need to ensure that the assistant provides relevant responses in SNEHA's program domains and avoids off-topic queries. This is crucial for maintaining the quality of interactions in real-world scenarios.

Describe the solution you'd like

  • Run the topic relevance guardrail from kaapi-guardrails on a randomly sampled subset of ~14,605 conversations of SNEHA.
  • Start with 100 conversations to refine the prompt, then 500, and finally ~1.5k (~10% sampling).
  • Document results since we don't yet have annotations or a known on-topic/off-topic ratio to stratify on.
Dominant language
Python
Stars
5
Forks
4
Avg merge
11d 19h
Merged PRs (30d)
1

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ProjectTech4DevAI/kaapi-guardrails

All issues in ProjectTech4DevAI/kaapi-guardrails

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.