Agent harness: rig-based loop, layered prompt system, and failure-driven prompt improvement

Aperta
  • Qualcosa non va?
#485 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
ai-infra-agents, backend

Direzione di ricerca

Start by reading the proposed build order, especially assemble_brief and syscribe brief, then inspect the existing MCP server tools and example models. The issue proposes a new syscribe-harness crate and several open design decisions, so first scope a single build-order step and identify its existing entry points and tests. Done criteria should come from an ADR, requirements, and PlanningItem tree, which the issue says still need to be created.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Summary

Syscribe already provides the state and governance layer of an agent harness: a PlanningItem work queue, claim/release, an MCP server with guarded writes, [agents] policy (protected_paths, actors), human-owned questions (answerBy: human), an audit trail, and "done" gates (validate, trace-code, E719, W310). What is missing is the agent loop and the prompt system around it.

Proposal: a new syscribe-harness crate in which a rig agent is an MCP client of the existing server, so every guarded-write check still applies, plus a layered prompt system that learns from its own failures.

Goals

  1. A loop that takes a PI-*, claims it, assembles context, runs a rig agent, runs the gates, and either marks the item done with evidence or records a question and releases the claim.
  2. A layered prompt system whose task context is generated from the model, not hand-written.
  3. A feedback loop that turns gate failures into reviewed, evaluated prompt improvements.

Design

Integration shape
  • A (recommended first): rig as an MCP client of syscribe mcp. Reuses guarded writes, dry-run/delta/commit, --expect-hash, protected paths.
  • B (later, if profiling justifies): depend on syscribe-model and register native rig Tools. Faster, but easy to bypass the guarded-write path.
Prompt layers (stable to volatile, so the prefix caches)
# Layer Source
1 Role and rules new prompts/harness/role.md, distilled from the CLAUDE.md "Safe mutation" rules
2 Format guide core of create-model.md plus on-demand spec <topic> / help <Type> lookup
3 Project overlay existing agent_instructions_section hooks (link types, binding time), [agents], [users]
4 Task brief for one PI-* new assemble_brief(model, pi); syscribe brief <PI> [--json] [--budget N]
5 Gate contract generated from the real gate commands

Templates live as files in prompts/harness/*.md with a slot delimiter that does not collide with {{X.y}} value references. A [harness] table in .syscribe.toml may override a layer.

The brief contains: the goal and parent/child state (steps), full text of every achieves: requirement with ancestors and breakdownAdr, satisfiedBy elements and implementedBy paths, existing TestCases and verdicts, answered/open questions and blockedBy, and hazards (W311 overlaps, suspect links, protected paths). Each section has a token budget; anything cut is listed as an id pointer.

Learning from failures
  1. Run record sidecar per run: prompt layer hashes, model id, PI id, brief hash, outcome (done | gate-failed | escalated | retry-exhausted | abandoned), gate findings per attempt, rejected writes, tool errors, cost.
  2. harness failures: deterministic aggregation by finding code and prompt layer, with a cause classification (instruction missing, ignored, brief missing context, unhelpful tool error, model limit).
  3. harness analyze: LLM-proposed diff to one layer, emitted as a PlanningItem with a falsifiable claim. Never a direct edit.
  4. Replay set and runner: baseline vs candidate, N runs each, gate-pass rate / retries / escalations / cost. One layer changed per proposal. Rules that never fire are pruning candidates.
  5. Human gate: approval through an answerBy: human question.

Tool calls the rig agent needs

Most exist in the MCP server today (names from tool_names()); gaps are marked.

Orient and read

Tool Purpose Status
get_element fetch an element (detail, fields projection, docRendered) exists
steps, questions PI breakdown, open questions exists
trace, impact, follow, neighbors traceability context exists
search, search_text, list_by_type, children locate elements exists
describe_type, explain_finding, template learn the format on demand exists
get_brief the assembled task brief new

Write (guarded)

Tool Purpose Status
next_id never invent an id exists
create_element, update_element, move_element, delete_element model edits exists
evidence attach evidence to a PI exists
question record a decision instead of guessing exists
claim_item, release_item ownership; claim/release are CLI-only today new (verify)
amend amendment.add on an accepted ADR / approved requirement verify via update_element

Verify and gate

Tool Purpose Status
validate, validate_element model checks exists
trace_code untraced code units exists
gates the "before declaring done" set exists, check it covers W310/E719
ingest_results feed test results back as evidence exists
requirement_coverage verdict per requirement exists

Code side (not in the MCP server)

Tool Purpose Status
read_file, edit_file source edits, honouring protected_paths new
run_command cargo test, cargo build, allow-listed new
finish the agent signals completion, which triggers the gate run new, harness-local
escalate raise an answerBy: human question and stop new, harness-local (wraps question)

finish and escalate are handled by the harness loop itself rather than the model server, so the harness, not the agent, decides when the work is done.

Build order

  1. assemble_brief plus syscribe brief, with golden tests on the example models.
  2. Embedded layer templates and harness prompt <PI> (print the assembled prompt).
  3. Trim create-model.md to a core plus on-demand lookup, measured first.
  4. rig loop: claim, brief, agent, gates, retry (capped), escalate, release.
  5. Run-record sidecar and harness failures.
  6. Replay runner.
  7. harness analyze, last.

Open decisions

  • Big cached format-guide prefix, or on-demand lookup? (lean: on-demand, then measure)
  • Slot delimiter for templates (must not collide with {{X.y}})
  • Do prompt files carry amendments: records?
  • Retry cap and which PIs an agent may take (exclude answerBy: human blockers)
  • Where run records live and whether transcripts are stored (privacy)

Process (per repo convention)

  • ADR covering layering, brief, rig-over-MCP, and prompt evolution
  • REQ-TRS-HARNESS-* requirements
  • Native PlanningItem tree under model/Planning/
  • Design element whose implementedBy: names the new crate (avoids W326)
  • Discoverability checklist: template, help, search, tasks, explain, agent-guide, docs
Lingua principale
Rust
Stelle
8
Fork
1
Merge medio
4h 44m
PR unite (30g)
51

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di sjames/syscribe

Tutte le issue di sjames/syscribe

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.