Owner

iris-eval

1 indexed repository · View on GitHub

mcp-server
The agent eval standard for MCP — score output quality, catch safety failures, enforce cost budgets

TypeScript · 9 stars

Beginner friendly Everything

Open All

no_tool_loop misses an N-call cycle for N >= 3 Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

iris-eval/mcp-server#427 · Opened Sep 4, 2026
Decision Moments: rank by significance, not only newest-first Open

acceptance-testing dashboard P2

Difficulty 4/5 3-5 days Newbie friendliness 50/100

iris-eval/mcp-server#409 · Opened Sep 3, 2026
Storage init: friendlier failure when better-sqlite3 bindings are missing (#369 item 7) Open

P3 server

Difficulty 3/5 1-2 days Newbie friendliness 72/100

iris-eval/mcp-server#408 · Opened Sep 3, 2026
verify_citations: judge failures are filed under resolve_error on citations whose resolve_status is ok Open

acceptance-testing P3 server

Difficulty 3/5 1-2 days Newbie friendliness 68/100

iris-eval/mcp-server#407 · Opened Sep 3, 2026
eval_type "all": a category with nothing evaluated reads as failing (passed:false, score:0, insufficient_data:true) Open

acceptance-testing P2 server

Difficulty 4/5 3-5 days Newbie friendliness 48/100

iris-eval/mcp-server#406 · Opened Sep 3, 2026
Input validation: nested objects and dashboard routes still strip unknown keys Open

acceptance-testing bug dashboard effort: large P1 security server

Difficulty 4/5 3-5 days Newbie friendliness 52/100

iris-eval/mcp-server#376 · Opened Aug 12, 2026
`passed` semantics: persistence and cross-tool consistency gaps after the critical-rule veto Open

acceptance-testing bug dashboard effort: large P1 server

Difficulty 4/5 3-5 days Newbie friendliness 52/100

iris-eval/mcp-server#375 · Opened Aug 12, 2026
PII patterns are label-anchored more than the pattern list implies Open

acceptance-testing bug docs effort: small P2 security server

Difficulty 3/5 1-2 days Newbie friendliness 68/100

iris-eval/mcp-server#374 · Opened Aug 12, 2026
Dashboard/API correctness batch Open

acceptance-testing bug dashboard docs effort: large P1 server

Difficulty 5/5 Over a week Newbie friendliness 35/100

iris-eval/mcp-server#373 · Opened Aug 12, 2026
Data retention and permissions for stored eval/trace text Open

acceptance-testing bug docs effort: medium P1 security server

Difficulty 5/5 Over a week Newbie friendliness 35/100

iris-eval/mcp-server#372 · Opened Aug 12, 2026
Shipped README: dead links, stale claims, and gaps found by install-only acceptance testing Open

acceptance-testing bug docs dx effort: medium P1

Difficulty 4/5 3-5 days Newbie friendliness 45/100

iris-eval/mcp-server#369 · Opened Aug 12, 2026
Upgrade root TypeScript to v6 when @typescript-eslint supports it Open

dependencies

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

iris-eval/mcp-server#58 · Opened Mar 23, 2026
Upgrade website ESLint to v10 when eslint-config-next supports it Open

dependencies

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

iris-eval/mcp-server#54 · Opened Mar 23, 2026
Add full-text search to trace queries Open

dashboard effort: large enhancement good first issue

Difficulty 5/5 Over a week Newbie friendliness 35/100

iris-eval/mcp-server#7 · Opened Mar 16, 2026
Create Python SDK for HTTP transport Open

effort: large enhancement good first issue

Difficulty 5/5 Over a week Newbie friendliness 35/100

iris-eval/mcp-server#6 · Opened Mar 16, 2026
Add webhook notifications for evaluation failures Open

effort: medium enhancement good first issue

Difficulty 4/5 3-5 days Newbie friendliness 35/100

iris-eval/mcp-server#5 · Opened Mar 16, 2026
Add CSV/JSON export to dashboard Open

dashboard effort: medium enhancement good first issue

Difficulty 4/5 3-5 days Newbie friendliness 45/100

iris-eval/mcp-server#4 · 2 comments · Opened Mar 16, 2026

iris-eval

Get new issues in your inbox