Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[REC-AR6][P0] Deterministic Recovery MVP Evaluation and Go/No-Go (Advisor Optional)

Aperta
  • Qualcosa non va?
#375 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai-infra-agents, security, testing-qa

Direzione di ricerca

Start by reviewing the REC-AR0..AR5 dependency issues and the security gate issues #318, #323, and #306. Define reproducible fake-E2E and adversarial evaluations covering the listed recovery cases, fixed models and budgets, crash/restart, offline effects, and independently verified outcomes. Done means tests preserve existing BLOCKED behavior and authority gates, report a go/no-go, and never run paid evaluations without explicit approval.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Parent: #354 · Stage: MVP M2 · Priority: P0 (deterministic evaluation) · Do not gate on optional Advisor issues.

Goal

Evaluate the existing Controller + #369 typed stop reasons + #374 deterministic escalation package against current operator experience. The EPIC can close its deterministic MVP with explicit evidence even if both #370 Advisor gates are no-go.

Study and delivery

  1. Preregister an offline/evidence review baseline including the review report's reported 51 runs / 15 unblocks (not independently verified here), fixed #291/#327 regressions, true K9/K10/manual coordination, crash recovery, LOCAL blocked, FAILED, unchanged-phase errors, timeouts and non-AutoForge samples where available.
  2. Define gold-standard operator resolution using Controller live readback or separately reviewed human labels, distinguish actual automated recovery from merely providing guidance, and never count Agent/Advisor confidence as outcome proof.
  3. Compare original status/errors against deterministic typed classification + action package. Measure classification accuracy/coverage, actionable handoff coverage, incorrect/unsafe suggested actions, human diagnosis time/steps, repeat BLOCKs, preserved candidates and effects, latency and added runtime cost. Include uncertainty, missing samples and selection bias.
  4. Required security regressions: zero unauthorized GitHub writes/unblock/approvals; zero lost unpushed commits or lost pending effect receipts; no bypass of exact-candidate validation, ReviewCertificate, CI, merge/ownership locks; no altered default Controller decisions; crash and JSON backward-compatibility tests.
  5. Record independent deterministic MVP go/no-go and operator sign-off. Advisor diagnosis shadow comparisons, if Gate D approves, belong to a separate follow-up evaluation Issue, not a blocker of this task. Autonomous execution, if Gate A approves, likewise requires separately reviewed safety rollout.
  6. No real paid-model calls, changed CI thresholds or autoforge epic run merely for planning. Default-on changes require their own approval even if this evaluation passes.

Acceptance criteria

  • Reproducible and annotated before/after matrix showing objective handoff effectiveness, not only automated unblock percentage.
  • Safety/correctness and crash/unknown-effect regression tests pass; unknown/ambiguous stays explicit.
  • #370 Gate D and A decisions recorded (including no-go), with optional #371–#373 not required for MVP closure.
  • Maintainer-approved roll-forward/rollback or explicit no-go with remaining risks and docs/status guidance.

Dependencies

Hard: #369, #370 and #374. No hard dependency on #371, #372 or #373, and no blanket #315/#319/#320/#321 gate. Coordinate #318/#323 evaluation conventions and #306 security boundaries.

Non-negotiable invariants

  • Controller-only authority for phase/task transitions, typed GitHub effects, candidate/evidence identity, authorization, validation, ReviewCertificate and merge; operator-only K9/K10 semantics unchanged. Existing BLOCKED and FAILED exit rules unchanged, LOCAL never gains an unblock edge by implication.
  • No provider receives GitHub mutation credentials, no prompt/source text is authorization, no untrusted shell commands outside existing sandbox policy. Preserve unpushed FIX candidates, effect journals and original evidence; reconcile UNKNOWN outcomes before retrying.
  • No implementation or paid-model benchmark is authorized by this planning change; default single-agent behavior unchanged. Persisted schema migrations require coordination on #23, using the protocol current at implementation time rather than reserving a number.
  • Native GitHub links remain to be established and read back: prose dependencies are not scheduler-enforceable blocked_by edges.

Review rationale: #354 review report.

Lingua principale
Python
Stelle
0
Fork
0
Merge medio
5h 16m
PR unite (30g)
100

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di iceboundrock/AutoForge

Tutte le issue di iceboundrock/AutoForge

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.