Owner

edonadei

1 indexed repository · View on GitHub

caliper
Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports C...

Python · 181 stars

Beginner friendly Everything

Open All

validate accepts a spec with no tasks, or with duplicate task names Open

area:spec bug good first issue ready-for-agent

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

edonadei/caliper#168 · Opened Sep 24, 2026
A timed-out hermes attempt still loses its transcript Open

area:harness bug ready-for-agent

Difficulty 3/5 1-2 days Newbie friendliness 65/100

edonadei/caliper#163 · Opened Sep 24, 2026
MCP preflight rejects servers that only speak 2026-07-28 Open

area:mcp enhancement question

Difficulty 3/5 1-2 days Newbie friendliness 65/100

edonadei/caliper#161 · Opened Sep 24, 2026
Sandbox judge-written scripts while preserving script mode Open

area:judge area:security enhancement

Difficulty 4/5 3-5 days Newbie friendliness 40/100

edonadei/caliper#158 · Opened Sep 24, 2026
Move in-repo specs off shared /tmp paths Open

area:spec bug good first issue ready-for-agent

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

edonadei/caliper#136 · Opened Sep 23, 2026
Jev-backed classify: findings, API and next steps (parked) Open

area:judge enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

edonadei/caliper#126 · Opened Sep 23, 2026
Make backend smoke evals portable on Windows Open

area:spec bug ready-for-agent

Difficulty 3/5 1-2 days Newbie friendliness 70/100

edonadei/caliper#120 · Opened Sep 22, 2026
Windows: isolated attempts crash and codex reads the real account Open

area:harness area:security bug ready-for-agent

Difficulty 3/5 1-2 days Newbie friendliness 65/100

edonadei/caliper#112 · Opened Sep 17, 2026
Spec naming convention assumes a subject the model no longer has Open

area:spec documentation question

Difficulty 2/5 1-3 hours Newbie friendliness 70/100

edonadei/caliper#82 · Opened Aug 3, 2026
Slash-command skills: invocation as a second loading discipline Open

area:spec enhancement question

Difficulty 4/5 3-5 days Newbie friendliness 45/100

edonadei/caliper#78 · Opened Aug 3, 2026
Classify judge API failures for codex / pi / hermes, one recorded fixture per backend Open

area:harness area:judge enhancement help wanted

Difficulty 3/5 1-2 days Newbie friendliness 65/100

edonadei/caliper#77 · Opened Jul 26, 2026
Report uncertainty: per-task confidence intervals on the score Open

area:scoring enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

edonadei/caliper#74 · Opened Jul 19, 2026
Add a pre-registered gate: declared bar + nonzero exit on failure Open

area:scoring enhancement question

Difficulty 4/5 3-5 days Newbie friendliness 45/100

edonadei/caliper#73 · Opened Jul 19, 2026
MCP support: emit a redacted MCP materialization receipt into the results JSON Open

area:mcp enhancement

Difficulty 3/5 1-2 days Newbie friendliness 55/100

edonadei/caliper#52 · 1 reaction · Opened Jul 9, 2026
Multi-turn / user-simulator harness for interview skills Open

area:harness enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

edonadei/caliper#16 · 2 comments · Opened Jul 1, 2026

edonadei

Get new issues in your inbox