Add a reproducible retrieval benchmark with at least 30 queries

Open
  • Is something wrong?
#46 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript, node.js
Domain
ai-infra-agents, search

Research direction

Read AGENTS.md, docs/PROJECT_CONTEXT.md, and the prerequisite issues #45 and #40; their dependent contracts must land before implementation, and the design should be discussed with the maintainer first. Then inspect the synthetic reference corpus, retrieval docs, and existing authorization, safe-error, and synthetic-fixture patterns to identify the benchmark entry point. Done means at least 30 MK/EN/SQ queries with explicit normalization, expected evidence and deterministic metrics, plus documented baseline scores and limits; the stated release target is at least 90% top-five retrieval for answerable queries with valid source references.

Written by the indexing model from the issue text.

Description

area:backend area:docs priority:medium status:blocked type:feature

Part of #28 · Target: Release readiness / v1

Goal and scope

Define at least 30 realistic answerable/unanswerable retrieval queries over the synthetic corpus, expected evidence and deterministic metrics. Record baseline scores and include MK/EN/SQ cases with explicit supported normalization.

Implementation starting points

synthetic reference corpus, new benchmark command, retrieval docs. Follow AGENTS.md and docs/PROJECT_CONTEXT.md; reuse existing authorization, safe-error and synthetic-fixture patterns. New runtime dependencies need a documented reason.

Dependencies and readiness

Depends on #45, #40. Discuss the design before starting implementation; dependent contracts must land first.

Out of scope

No faculty credential requirement, private query telemetry or fabricated passing scores.

Acceptance criteria and test plan

At release, at least 90% of answerable queries retrieve expected evidence in top five and all results have valid source references. Unanswerable/partial-scope cases remain explicit. Benchmark fails clearly on regressions and documents limits rather than claiming general AI accuracy.

Run the relevant offline tests, npm run check and git diff --check; runtime changes also require npm test and npm audit. Report checks not run. No faculty credentials or private course material are required for a contribution.

Contributing

Comment here before starting so the maintainer can coordinate the approach and avoid duplicate work. Fork the repository, branch from master, and open a focused PR referencing this issue. No assignee or delivery deadline is set; required checks and review apply.

Dominant language
JavaScript
Stars
0
Forks
0
Avg merge
1h 25m
Merged PRs (30d)
13

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mahmutmft/courses-mcp

All issues in mahmutmft/courses-mcp

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.