[DEL-A3] Reproducible baseline and predeclared Memory and multi-agent evaluation

Begin from accepted foundations #158/#176 and ADR 0004 to understand controller-owned effects and candidate validation. Use deterministic fake provider with existing CLI integration to build opt-in harness satisfying acceptance criteria. Done means baseline runs in reproducible local fixture mode with pinned manifest and complete opt-in smoke docs.

Parent: EPIC #315 · Stage M0 (architecture and baseline) · Size L · Reconciled 2026-10-25 (UTC) against main at f04db5f (state protocol 12). "Current scope" is authoritative. The earlier planning text follows verbatim and still applies where it does not conflict.

Current scope (reconciled 2026-10-25)

Build the evaluation harness and pre-register its criteria, so it is usable from the beginning. Running benchmarks against real models is a separately approved, opt-in act, never part of planning or CI.

First comparison: Memory benefit

Compare existing AutoForge without Memory against existing AutoForge with bounded Issue Working Memory (#317 + #333). Hold equal: repository tasks, base commits, models and providers, sandbox policy, tools, validation commands, review, and resource limits.

Measure:

• independently validated Issue completion;
• Review/FIX cycles;
• repeated failed approaches;
• duplicate exploration;
• human BLOCKED interventions;
• context tokens;
• wall-clock time;
• cost per validated success;
• crash-recovery correctness;
• stale-memory leakage;
• Memory overhead (storage, projection size, latency).

Second comparison: multi-agent benefit

Once the features exist, four arms:

1. single agent without Memory;
2. single agent with Issue Memory;
3. deterministic fixed-role multi-agent with Issue Memory;
4. adaptive Manager/Worker with Issue Memory.

Never compare the multi-agent system only against an artificially weakened single-call baseline. Arms 1 and 2 keep the existing validation and independent review.

Deterministic control-plane tests

Offline, with fakes:

• replay consistency;
• task-state transition correctness;
• duplicate-execution prevention;
• duplicate external-effect prevention;
• budget recovery after a crash;
• interrupted-launch reconciliation;
• candidate preservation;
• provider capability mismatch;
• stale-context denial;
• agent isolation;
• unknown-outcome handling.

Method

• Predeclare the go/no-go thresholds, non-regression margins, sample sizes and cost ceilings before any real run.
• Distinguish test-harness correctness (fakes) from measured model-quality change (real runs), and real outcomes from estimates.
• Distinguish "no validation configured" from "validation passed". Validation commands are empty by default (validation.commands).
• Report failures, interventions, unknown usage and per-task uncertainty, not only successes.

Dependencies

• Hard: none; it can start now.
• Blocks (native): #321, #323.
• Coordination: #154 (usage provenance); #333 and #319, which the later arms need. A missing feature leaves its arm "not evaluated"; it never blocks this issue.

Acceptance criteria (added 2026-10-25)

• Both comparisons and the control-plane test list are specified, with pinned tasks, base SHAs and equalized settings.
• Decision criteria are pre-registered before any real run.
• The no-memory baseline runs reproducibly in fake/local fixture mode.
• Reports separate harness correctness from model-quality evidence, and real outcomes from estimates.
• No benchmark runs during planning, and none runs in CI with real credentials.

Earlier planning text (verbatim)

Superseded where it conflicts with "Current scope" above. The memory-dependent and memory-independent task split below remains valid for the first comparison. The retrieval metrics (Recall@K, MRR) and offline-distillation costs apply to EPIC #331's optional evaluation, not to the Memory MVP comparison.

Parent: EPIC #315 · Wave A · Size L · Planning baseline f4fd5ef (2026-10-25 UTC). Research: GVS5H, not an implementation dependency.

Objective / deliverable

Design and implement opt-in test harness for representative AutoForge software-engineering issues, not only LiveCodeBench puzzles. Preserve a fixed benchmark set, expected base SHAs and independent acceptance tests, comparable provider/tool/sandbox configurations; classify infra failures, inaccessible credentials, truncation and skips. Instrument baseline single-agent quality, verified commits, Review/Fix, human interventions, elapsed time, tokens and cost estimates with provider provenance (coordinate #154). No model credentials or real GitHub writes in baseline tests; live runs need disposable repo and operator opt-in. Pre-register decision criteria before experimenting.

Dependencies

• Hard: none (can start from accepted #158/#176 foundations).
• Coordination only: #147 SDK providers / #126 interactions, #154 usage, #176 scheduler and #266 sandbox. Do not add blanket cross-EPIC hard blockers without evidence.

Acceptance criteria

• Baseline dataset manifest, deterministic fixture runner and comparison protocol are reviewable and pinned
• Collect validated PR or independent test success, review/fix rounds, BLOCKED, wall time, cost per verified success, unknown usage and safety failures
• No hidden-test leakage, benchmark issue contamination or cherry-picking; distinguish tasks and independent seeds
• Outcomes separately identify timeouts, costs, token uncertainty and missing external service availability
• At least single-agent baseline runs in reproducible fake/local fixture mode; real opt-in smoke documentation is complete
• Regression tests preserve original single-agent path, ADR 0004 controller-owned effects, exact candidate validation, ReviewCertificate and merge safety; provider credentials remain isolated.
• Docs/doctor/config compatibility updated where behavior changes; skipped real-runtime evidence never reported as pass.

Invariants / exclusions

The LLM Manager is advisory. Controller exclusively owns done acceptance, typed GitHub effects, state transitions, candidate identities and merge. Inner tasks are not GitHub native sub-issues and do not modify EPIC graph. Neither Agent nor verifier may execute repository-controlled commands outside the existing sandbox. No unbounded raw transcript/reasoning recording or ambient credentials. Feature remains opt-in; no Public Alpha #255 gate. Coordinate persisted schema on #23 before migration. Do not copy GVS5H's arbitrary subprocess verifier or its text-hash scratchpad and call it production recovery.

Verification hints

Use deterministic fake provider plus real existing CLI integration where available; include redacted bounded evidence, attack cases, failure/cancel/resume, explicit run/issue/candidate SHA. No new orchestration daemon or scheduler.

MemCurrent-inspired memory-dependent benchmark coverage (2026-10-25)

• Add pinned memory-dependent real software-engineering tasks (an earlier verified workaround materially helps) and matched memory-independent controls. Track whether recall actually changes verified outcomes, not only hit rates.
• Test obsolete/current/conflicting/corrected memories, volatile GitHub state, scope boundaries, negative debugging lessons, hallucinated lesson candidates, prompt-injection strings and incomplete index watermarks.
• Pre-register retrieval metrics (Recall@K/MRR, stale-fact/permission leak), successful independently validated PRs, Review/Fix rounds, human BLOCKED interventions, time-to-success, token and all-in cost per validated success, including offline distillation, index builds, human review and retrieval.
• Distinguish Databricks internal business-agent benchmark from AutoForge software-repository experiments and never treat reported 22.3%/68.4% or token savings as expected AutoForge performance.