Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Add a local, offline eval harness for Michael's routing and memory recall

Open
  • Is something wrong?
#53 2 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@agentvivekkumar is already working on this.

Since Oct 10, 2026.

Assessment

This issue has not been assessed yet.

Description

enhancement Require Customer
Problem / motivation

Don't Be Michael asks a small-business owner to trust one decision above all others: Michael routes every job to the right team member.

That routing is prompt-driven. Michael (the GOD agent) owns the roster in registry.json, adjudicates traffic, is the single scribe for board.md, and owns the task ledger in tasks.json. HIVE.md positions his escalation policy as a prompt-level control surface.

That makes the prompt powerful, but today there's no regression check for it:

  • Change Michael's system prompt, a role definition, an Office Pack in resources/packs/, or the hire-check logic, and there's no repeatable way to know if routing got better or worse.
  • A task that should have gone to Finance can silently go to Support, with the wrong mailbox and wrong private folder as a consequence (folder access follows the assignee).
  • Memory recall has the same gap. memory.ts wraps MemPalace with a clean degrade-to-noop path when it isn't installed, but there's no measured comparison of markdown-only memory vs MemPalace-backed recall on the same queries.
  • The existing tests in test/ protect code paths. They don't protect judgement — and judgement is the product.

For a non-technical owner, that's invisible until a real customer email lands with the wrong person.

Proposed solution

Add a small, local, offline eval harness for judgement — no network, no telemetry, no new runtime dependency.

  1. Tiny curated fixture set in test/eval/fixtures/ (~30 tasks from the Office Packs) each with task, expectedAssigneeRole, expectedEscalation (autonomous/needs-owner), mustNotAccess, and optional memoryQuery.
  2. Deterministic runner: npm run eval:routing (node:test style like test:focused) spinning a temp harnessHome, stubbed judge for offline CI, scoring Routing accuracy with per-role confusion list and Escalation calibration (over/under-escalation rates). When MemPalace is present, also report recall@k markdown-only vs MemPalace-backed; when absent, skip and say so.
  3. Local JSON report + 5-line plain-language summary. Possible v2: readout in Michael's Advanced/Monitor tab next to telemetry.ts/usage.ts.

Suggested scope: src/shared/evalFixtures.ts schema, runner under test/eval/ reusing hive.ts temp-home, npm script, local eval-report.json. No new DB, no new dependency.

Why it matters: Michael is the only one who assigns work. If routing drifts, folder privacy, mailbox permissions and Send-on-approval are applied faithfully to the wrong person. An eval harness turns prompt tuning into a number for PRs and a sentence an owner can understand. It also safety-nets #48 provider work.

Acceptance criteria:

  • npm run eval:routing runs fully offline and deterministically
  • Fixtures cover 3+ Office Packs with autonomous + needs-owner + must-not-access cases
  • Report separates routing accuracy from escalation calibration
  • MemPalace absent -> memory leg skips cleanly with note
  • No owner data read, no network, no telemetry during run
  • Swapping two roles in fixtures visibly drops the score
Alternatives considered

Manual QA checklist — doesn't scale with weekly prompt/pack changes, not reviewable in PR diff. Live A/B on real owners — conflicts with local-first/no-telemetry trust model. Full LLM-as-judge with paid API per run — non-deterministic, costly, breaks offline promise (stubbed judge + optional live flag is better). Rely only on #51/#47 — those verify a single task finished well; this measures whether assignment itself got better or worse after a change.

Area

The hive (memory, routing, GOD agent)

Pre-flight
  • I searched existing issues and discussions, and this isn't a duplicate.
Dominant language
TypeScript
Stars
273
Forks
211
Avg merge
9h 17m
Merged PRs (30d)
162

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from agentvivekkumar/dontbemichael

All issues in agentvivekkumar/dontbemichael

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.