Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Tool admission criteria and evaluation harness for PLATE MCP tools

Open
  • Is something wrong?
#1,198 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
devtools

Research direction

Start by running python -m plate_core.mcp_server to inspect the current tool inventory, then read the related issues #1192, #1126, and #1040 referenced here. The issue names docs/plate/research/tool-evals/ as the scorecard location and sketches the harness arms and pass bar. Done means the draft criteria and pass bar are confirmed and the harness plan is turned into concrete child tasks; the issue does not identify implementation files or tests.

Written by the indexing model from the issue text.

Description

area:agent Feature need:refinement

Summary

On 2026-10-23 Andrew asked for "some kind of evaluation harness for new tools, and … some clear criteria for which tools belong in PLATE and which clearly do not."

Why now. plate-mcp tools/list returns 202 tools. I ran python -m plate_core.mcp_server at main 238edc5. The schemas total about 85k characters, roughly 21k tokens, which a client that loads every tool pays each session. #1192 found that, in Vesalius's testing, Claude used a server's tools in only 2 of 24 runs when it exposed 20 tools. With 5 tools (2 loaded up front) it used them in every run. Nothing today says when a tool should be added or removed, and no harness measures whether a tool helps.

This issue defines draft admission criteria, sketches a cheap harness, and gives a first-pass verdict on today's tools (appendix).

1. Admission criteria (DRAFT, for Andrew to confirm)

Belongs in PLATE (every item must hold):

  • It enforces or automates a PLATE process rule (labels, merge gates, Q&A, release ceremony, stubs) that an agent would otherwise get wrong.
  • It needs repo or process state an agent can't get cheaply otherwise: several API calls, joins, or PLATE-local state such as ledgers and .agentic/. It returns a compact answer.
  • It's deterministic. Reads are side-effect free, and writes are idempotent with a dry run.
  • It shows a measured win in the harness (section 2) and gets called when it's relevant.
  • Its name is distinct and its schema is small (about 150 tokens or less).

Clearly does not belong:

  • A thin wrapper over a single gh/git command or a GitHub MCP tool (for example list or get discussions).
  • General engine or domain knowledge. That belongs in skills or external packs (see #1184).
  • A duplicate or near-duplicate of another tool, such as parallel state machines with the same verbs. Merge these behind a kind parameter.
  • Unused: not called in N relevant harness runs, or no real calls in 30 days of usage telemetry (#1192).
  • Long-running orchestration that fits better as a CLI command or scheduled workflow than as an interactive tool.
  • One-time setup (adopt, migrate, config). Keep it CLI-only and don't load it by default.

Tiers:

  • Core: about 5–8 tools, always loaded.
  • On-demand: deferred behind a profile or tool search.
  • CLI-only.
  • Cut.

2. Harness design sketch

  • Task set: 20–30 real PLATE agent tasks, each with an answer key or a checkable end state, plus a held-out set. Draw them from the #1126 issue replays and the #1040 process queries. Examples:

    • "What should I work on next?"
    • "Is PR #N mergeable under PLATE gates?"
    • "File a blocking Question."
    • "Which Features block Epic #N?"
    • "Repair release tracks (dry run)."
    • "Address the review threads on PR #N."

  • Arms: run each task four ways.

    • A: no PLATE MCP, only gh and the CLI.
    • B: the full toolset (202 tools).
    • C: the small core default.
    • D: C plus the candidate tool.

  • Metrics:

    • task success, graded blind against the answer key, or a deterministic end-state check
    • tokens, including tool-schema overhead
    • turns, wall time and cost
    • tool-call rate: was the candidate called when relevant?
    • wrong-tool calls and PLATE rule violations

  • Stats: use the Vesalius method: geometric mean of per-task ratios with bootstrap confidence intervals. A small n is fine at first.

  • Pass bar (DRAFT): a tool is admitted to on-demand if, comparing D with C:

    • success is no worse (CI lower bound ≥ −2 points)
    • and it gives at least 15% fewer tokens, or at least 1 fewer turn, or higher success
    • and it's called in at least 70% of the tasks where it's relevant

    Core additionally needs use in at least 30% of all tasks and a schema of about 150 tokens or less.

  • Cheap by design:

    • one run per task per arm
    • a per-run budget cap
    • a frozen sandbox repo fixture, so writes are safe and repeatable
    • a local-model arm on the MacBook (qwen3-coder:30b via the #1170 setup) for smoke runs, with cloud runs only for admission decisions

  • Output: a per-tool scorecard under docs/plate/research/tool-evals/. New plate_* tools need a scorecard before they merge.

Proposed children (not filed yet)

  • Research: harness task set, answer keys and held-out split (reuses #1126 and #1040).
  • Feature: harness runner with arms A–D, metrics and a report, plus a local-model option (#1170).
  • Feature: MCP tool tiers and profiles (core default, on-demand, CLI-only), for example PLATE_MCP_PROFILE.
  • Audit: run the harness on today's 202 tools and propose consolidation. The obvious starting points are 13 *_feed tools, about 35 list/get tools, and 4 parallel planning-session families.
  • Feature: local MCP usage tally (already proposed in #1192).
  • Task: a CI admission gate. A PR adding a plate_* tool must include a scorecard and a criteria checklist.
  • Question: Andrew confirms the criteria and the pass bar.

Part of #1121 (harness alignment). Related: #1192 (the tool-count finding), #1126 (issue-replay benchmark), #1040 and #1047 (token measurement), #1044 (new KG tools should go through this gate), #1010, #1170, #1068 (CLI and MCP reference).

Appendix: current tools by group (first pass)

The verdicts below apply to groups. Per-tool verdicts come from the Audit child.

Group # Tools First-pass verdict
Core orientation 15 health, what_next, epic_status, feed, features, context(s), agent(s), skill(s), costs, contemplate, synthesize_priorities, perform_information_audit Keep core for health, what_next, epic_status and feed. The rest are on-demand.
PR review and merge gates 4 get_pr_merge_gates, get_actionable_review_threads, resolve_review_thread, pr_babysit Keep core for gates and actionable threads. resolve_review_thread is a thin wrapper, so on-demand.
Questions and Q&A 5 list_questions, get_question, get_answers, record_answer, create_blocking_question Keep core for create_blocking_question and record_answer. The rest are on-demand.
Discussions and ideas 7 list/get/create_discussion, list_discussion_comments, add_discussion_comment, list_discussion_categories, list_open_ideas Likely cut: thin GitHub wrappers. Keep list_open_ideas on-demand.
Issue authoring 11 stub_* (7), task_create/close/detect, plan_epic On-demand. Consolidate the overlapping stub_author, stub_author_create and stub_create.
Release 5 release_status, release_repair, release_target_epic, release_notes, release_cleanup_branches On-demand.
Release media 5 release_media_* Likely cut, or merge into the evidence group.
Spec audit 2 spec_audit, spec_audit_followups On-demand.
Adoption, bootstrap, config, payload, migration 21 bootstrap, import_payload, adoption_*, config_*, payload_*, migrate_*, self_migrate_*, init_playwright CLI-only: one-time setup.
Planning sessions 33 planning_* (8), er_planning_* (7), design_contract_* (8), hybrid_* (10) Likely cut and consolidate: four parallel state machines with the same verbs (start, answer, build, script, decide, resubmit, list_pending).
Feature and bug loops 17 feature_loop_* (9), bug_loop_* (8) On-demand, merged into one loop_* set with a type parameter.
Evidence, media and packaging 18 feature_media_* (9), packaging_* (7), record_e2e_gif, validate_e2e_tests Likely cut or CLI-only. Overlaps gh --attach and #1051.
Orchestration 39 pm_* (8), fleet_* (9), collab_* (8), scheduled_op(s)_* (7), autonomy_* (6), delegate_to_agent On-demand behind an orchestrator profile only. Anything unused gets cut.
Approval and state stores 15 artifact_* (6), checkpoint_* (4), ledger_* (5) On-demand. Consolidate the overlapping propose and decide flows.
Monitoring 5 monitor_* Likely cut: these fit better as scheduled ops than interactive tools.
Total 202 Core about 8, and everything else deferred, CLI-only or cut.
Dominant language
Python
Stars
1
Forks
0
Avg merge
2d 10h
Merged PRs (30d)
39

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from akasper/plate

All issues in akasper/plate

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.