Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[REC-AR6][P0] Deterministic Recovery MVP Evaluation and Go/No-Go (Advisor Optional)

Open
  • Is something wrong?
#375 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
ai-infra-agents, security, testing-qa

Research direction

Start by reviewing the REC-AR0..AR5 dependency issues and the security gate issues #318, #323, and #306. Define reproducible fake-E2E and adversarial evaluations covering the listed recovery cases, fixed models and budgets, crash/restart, offline effects, and independently verified outcomes. Done means tests preserve existing BLOCKED behavior and authority gates, report a go/no-go, and never run paid evaluations without explicit approval.

Written by the indexing model from the issue text.

Description

Parent: #354 · Stage: MVP M2 · Priority: P0 (deterministic evaluation) · Do not gate on optional Advisor issues.

Goal

Evaluate the existing Controller + #369 typed stop reasons + #374 deterministic escalation package against current operator experience. The EPIC can close its deterministic MVP with explicit evidence even if both #370 Advisor gates are no-go.

Study and delivery

  1. Preregister an offline/evidence review baseline including the review report's reported 51 runs / 15 unblocks (not independently verified here), fixed #291/#327 regressions, true K9/K10/manual coordination, crash recovery, LOCAL blocked, FAILED, unchanged-phase errors, timeouts and non-AutoForge samples where available.
  2. Define gold-standard operator resolution using Controller live readback or separately reviewed human labels, distinguish actual automated recovery from merely providing guidance, and never count Agent/Advisor confidence as outcome proof.
  3. Compare original status/errors against deterministic typed classification + action package. Measure classification accuracy/coverage, actionable handoff coverage, incorrect/unsafe suggested actions, human diagnosis time/steps, repeat BLOCKs, preserved candidates and effects, latency and added runtime cost. Include uncertainty, missing samples and selection bias.
  4. Required security regressions: zero unauthorized GitHub writes/unblock/approvals; zero lost unpushed commits or lost pending effect receipts; no bypass of exact-candidate validation, ReviewCertificate, CI, merge/ownership locks; no altered default Controller decisions; crash and JSON backward-compatibility tests.
  5. Record independent deterministic MVP go/no-go and operator sign-off. Advisor diagnosis shadow comparisons, if Gate D approves, belong to a separate follow-up evaluation Issue, not a blocker of this task. Autonomous execution, if Gate A approves, likewise requires separately reviewed safety rollout.
  6. No real paid-model calls, changed CI thresholds or autoforge epic run merely for planning. Default-on changes require their own approval even if this evaluation passes.

Acceptance criteria

  • Reproducible and annotated before/after matrix showing objective handoff effectiveness, not only automated unblock percentage.
  • Safety/correctness and crash/unknown-effect regression tests pass; unknown/ambiguous stays explicit.
  • #370 Gate D and A decisions recorded (including no-go), with optional #371–#373 not required for MVP closure.
  • Maintainer-approved roll-forward/rollback or explicit no-go with remaining risks and docs/status guidance.

Dependencies

Hard: #369, #370 and #374. No hard dependency on #371, #372 or #373, and no blanket #315/#319/#320/#321 gate. Coordinate #318/#323 evaluation conventions and #306 security boundaries.

Non-negotiable invariants

  • Controller-only authority for phase/task transitions, typed GitHub effects, candidate/evidence identity, authorization, validation, ReviewCertificate and merge; operator-only K9/K10 semantics unchanged. Existing BLOCKED and FAILED exit rules unchanged, LOCAL never gains an unblock edge by implication.
  • No provider receives GitHub mutation credentials, no prompt/source text is authorization, no untrusted shell commands outside existing sandbox policy. Preserve unpushed FIX candidates, effect journals and original evidence; reconcile UNKNOWN outcomes before retrying.
  • No implementation or paid-model benchmark is authorized by this planning change; default single-agent behavior unchanged. Persisted schema migrations require coordination on #23, using the protocol current at implementation time rather than reserving a number.
  • Native GitHub links remain to be established and read back: prose dependencies are not scheduler-enforceable blocked_by edges.

Review rationale: #354 review report.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6h
Merged PRs (30d)
78

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from iceboundrock/AutoForge

All issues in iceboundrock/AutoForge

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.