feature: add bounded verification and repair to Router Flow
#2,867 opened on 2026/08/11
Repository metrics
- Stars
- (4,293 個のスター)
- PR merge metrics
- (PR metrics pending)
説明
Summary
Add explicit verification, bounded repair, and context compaction to Router Flow while preserving workflow state ownership, per-step controls, and hard request budgets.
Parent hardening tracker: #2336
Blocked by: #1456, #1815, #2471, #2855, #2857, #2859, #2861
Current behavior and gaps
Workflows supports static roles or a dynamic planner, sequential steps with parallel models inside a step, final synthesis, and stateful tool pause/resume. However:
verifier,reviewer, andcriticare prompt roles rather than typed checks with scores/results;- failed or weak outputs do not enter a generic verify-repair loop with an explicit budget;
- accessible prior step outputs are repeatedly embedded in later prompts and can grow context quickly;
- steps execute sequentially even when dependencies could eventually permit independent scheduling;
- production tool-resume semantics depend on the state lifecycle/durability work in #2471;
- stable per-step timeout/token/output receipts belong with #1815.
Proposed phases
Typed verification
- Add workflow step/final verification configuration backed by #2857.
- Prefer deterministic signals: parser/schema validation, executable tests, sandbox results, tool-result contracts, or trusted-source checks.
- Use LLM judges only as calibrated outcome verifiers, not as unbounded free-form critics.
Bounded repair
- On a failed verification result, generate a targeted repair request containing the original contract, bounded failure evidence, and permitted prior outputs.
- Limit attempts, tokens, cost, calls, and wall time via #1456/#2861.
- Define fail, skip, fallback, and abstain outcomes when repair is exhausted.
- Persist pause/resume state and exactly-once tool turns through #2471.
Context compaction
- Add per-step output summaries or bounded reference views instead of replaying every full prior response.
- Preserve output-contract fields and trusted tool/test evidence losslessly.
- Record before/after token estimates and omitted references.
Dependency-aware scheduling (later slice)
- Derive a DAG from explicit access/dependency declarations.
- Run only independent ready steps concurrently.
- Keep this behind static-plan validation and do not merge it into the first verification/repair PR.
Research context
Tool-verified execution is strongest where the environment supplies objective feedback, such as code tests, SQL execution, schemas, and agent tools. ICML 2025 JETTS cautions that natural-language critique alone is ineffective for guiding refinement; repair should therefore be anchored in typed or executable evidence whenever possible.
Non-goals
- Do not create a general external workflow engine.
- Do not bypass decision candidate sets, authorization, plugins, or tool policy.
- Do not duplicate #2471 state ownership or #1815 per-step accounting.
- Do not permit unbounded autonomous retries.
Acceptance criteria
- Static and dynamic plans can declare typed verifier and repair policies with validation.
- Deterministic tool/test/schema verification is supported end to end.
- Repair attempts are bounded and every verify/repair/tool turn is traced and budgeted.
- Exhaustion has deterministic fail/skip/fallback/abstain semantics.
- Context compaction reduces repeated prompt tokens without losing required contracts/evidence.
- Two-request tool pause/resume remains exactly once and durable under #2471 semantics.
- Per-step and aggregate receipts align with #1815 and #2855.
- Benchmark slices compare unverified, verified, and verify-repair Flow on executable agentic tasks.
- DAG scheduling, if implemented, lands as a later independently reviewable slice.