test(rag): retrieval hit-rate eval over golden questions
Maintainers usually reply within 1 day
@DerekRoberts is already working on this.
Since Oct 24, 2026.
Assessment
This issue has not been assessed yet.
Description
Problem
Vexilon has no measure of retrieval quality, so changes to chunking or search can't be checked. #714 (golden questions gate before PROD) is closed, but no golden questions file or gate turns up in the repo (grep -ri golden finds nothing).
Scope
- First find or restore the #714 golden questions gate.
- Write about 30 real steward questions. Label each one with its expected source document and section or article. Include section-number and defined-term questions, e.g. "s. 21 WCA" or "Article 32.04".
- Add a CI test that runs them through
search_indexand reports top-5 and top-10 hit rates against a committed baseline.
Acceptance criteria
- The golden questions file is in the repo, and CI reports top-5 and top-10 hit rates against a committed baseline, failing if they regress.
- Dominant language
- Python
- Stars
- 2
- Forks
- 1
- Avg merge
- 4h 16m
- Merged PRs (30d)
- 38
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from MinionTech/vexilon
-
Difficulty 2/5 Half a day Newbie friendliness 66/100
MinionTech/vexilon#844 ·
Maintainers usually reply within 1 day
-
OHS Part 8 source missing B.C. Reg. 135/2026 (face-seal amendments, in force 2026-11-06)Possibly taken @DerekRoberts claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
MinionTech/vexilon#842 · 1 assignee ·
Maintainers usually reply within 1 day
-
research: consume from canli.orgPossibly taken @DerekRoberts claimed this 1 day ago. Open
MinionTech/vexilon#836 · 1 assignee ·
Maintainers usually reply within 1 day
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 22/100
MinionTech/vexilon#821 · 1 assignee ·
Maintainers usually reply within 1 day
-
feat(rag): hybrid BM25 + FAISS retrieval with reciprocal-rank fusionMay be free again @DerekRoberts claimed this 28 days ago, and no pull request is open. Openenhancement
MinionTech/vexilon#802 · 1 assignee ·
Maintainers usually reply within 1 day
All issues in MinionTech/vexilon
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
cal-itp/data-infra#5887 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
ranacseruet/vaice#139 ·
Maintainers usually reply within 1 day
-
numpy.bytes_ attributes are recorded as a truncated repr instead of their contentPossibly taken @JoeyTan21 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
batocera-linux/batocera.linux#17423 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100