test(rag): retrieval hit-rate eval over golden questions

Open
#801 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@DerekRoberts is already working on this.

Since Oct 24, 2026.

Assessment

This issue has not been assessed yet.

Description

enhancement

Problem

Vexilon has no measure of retrieval quality, so changes to chunking or search can't be checked. #714 (golden questions gate before PROD) is closed, but no golden questions file or gate turns up in the repo (grep -ri golden finds nothing).

Scope

  1. First find or restore the #714 golden questions gate.
  2. Write about 30 real steward questions. Label each one with its expected source document and section or article. Include section-number and defined-term questions, e.g. "s. 21 WCA" or "Article 32.04".
  3. Add a CI test that runs them through search_index and reports top-5 and top-10 hit rates against a committed baseline.

Acceptance criteria

  • The golden questions file is in the repo, and CI reports top-5 and top-10 hit rates against a committed baseline, failing if they regress.
Dominant language
Python
Stars
2
Forks
1
Avg merge
4h 16m
Merged PRs (30d)
38

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MinionTech/vexilon

All issues in MinionTech/vexilon

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.