[DEL-C3] Offline Memory Consolidation Pipeline: checkpointed distillation and review proposals

Begin by reading issues #316, #317, #325 and #324 to understand the memory architecture and lifecycle. Locate the existing MemoryStore and Ledger code in the Python repo and the AgentProvider/Executor sandbox policy. 'Done' means the pipeline passes the deterministic fake-LLM test and crash matrix with no approved record changes before review.

Parent: EPIC #315 · Wave C (optional post-MVP memory maintenance) · Size L · Task: DEL-C3. Inspiration: MemCurrent design article (Databricks, 2026-10-20) and Databricks Agent memory pipelines. Native AutoForge implementation only; no Databricks service required.

Objective

Implement a replay-safe, bounded offline Memory Consolidation Pipeline that ingests new Controller-owned, sanitized Issue evidence / typed Ledger summaries, distills proposed reusable lessons, identifies duplicates/conflicts and suggests merge/split/supersede actions to #325's review-gated Repository Knowledge. The pipeline produces untrusted review candidates; it never unilaterally mutates or deletes approved knowledge, changes workflow state, installs skills or publishes GitHub effects.

Distinguish online immediate corrections in #317 from offline derived knowledge generation and maintenance in this task. MemCurrent's raw-session episodic capture and fully automatic mutable memory store are not authorized for AutoForge.

Hard dependencies

• #316 — accepted Memory architecture/authority and pipeline boundaries
• #317 — typed Ledger, evidence provenance, freshness/event sequence, safe incremental export
• #325 — typed proposed/approved/retracted/rejected Knowledge lifecycle and approval intake
• Transitive via #325: #324 safe repository-scoped retrieval/filtering for duplicate candidates. A pipeline must still function in a deterministic/no-semantic-index mode.

Non-blocking extension: Not a hard dependency of basic #319 fresh handoffs, #320 verifier feedback, #321 adaptive budget, #323 existing release/ablation gate, #255 Public Alpha, #266 Docker or #147 SDK. A dedicated future experiment can include it if available.

Functional contract

1. Inputs and classification: read approved-to-process, redacted/bounded structured records and Controller-verified evidence receipts from #317; use a source-key (repo id, issue id, run id, event seq, verified candidate SHA) and source validation. Do not ingest full raw transcripts, model hidden reasoning, secrets or cross-repository records by default. Prefer durable engineering knowledge (corrections, debugging recipes, repo conventions, repeated failure patterns), not volatile live GitHub PR state that host gh can re-query.
2. Stages: scan incremental sources → select → derive candidate lessons → detect overlap/conflicts → propose merge/split/supersede/expiry → stage typed proposals → submit to #325 review queue. Each proposed claim includes exact evidence refs, scope, applicability (tool/provider/platform/version/commit), freshness, confidence as advisory metadata, and explanation of the transformation. Never rewrite existing approved memory directly.
3. Checkpoints and identity: checkpoint per repository and per Issue/source partition, not globally, tracking successful-through event/verified SHA, input schema version, processing job ID, policy/model revision and hashes of emitted proposals. A failure for one partition does not lose/skip others. Only commit a checkpoint atomically with successful staging of bounded output. Resume is at-least-once with idempotency keys; duplicate/replayed runs cannot duplicate Knowledge candidates.
4. Read-before-consolidate: consult current authoritative MemoryStore and optional #324 retrieval for overlap, contradictions and applicability; stale index results or missing results never override a current retraction/correction. Conflicts unresolved by verified evidence become human-review questions; do not auto-resolve by text recency or model confidence.
5. Repair/maintenance: suggest merging duplicates, splitting oversized lessons, invalidating stale applicability, dropping low-signal/too-volatile candidate content and superseding outdated claims. Changes to approved entries must traverse #325's independent review and audit trail. Existing rejected/retracted knowledge must not resurrect after backfill, replay, or index rebuild.
6. Execution isolation: the orchestrator and checkpoint authority stay on trusted Host Controller; any LLM distillation runs through existing AgentProvider/Executor sandbox policy with provider-specific limited model credentials only. No unconfined host shell, no GitHub write permissions, no raw gh command proxy, no automatic third-party evaluator or upload. If a safe provider/sandbox is unavailable, defer/stop with explicit status, not an insecure fallback.
7. Trigger and controls: opt-in manual invocation first; any future periodic/on-success scheduling is explicitly configured, rate/cost/concurrency bounded and disable-able. Never promise an OS background daemon or enable autonomous uploading by default. Budgets include model inference, summarization, indexing, host CPU/IO and human review. Retention/purge and operator pause/resume are documented.
8. Interface/operations: idempotent preview/dry-run, run, status, retry and bounded audit/history; outputs name proposed candidate IDs, checkpoint watermarks, skipped/unverifiable events and costs without leaking prompts/credentials. Avoid inventing persistent schema numbers; coordinate with #23.

Acceptance criteria

• Deterministic fake-LLM/no-network test drives incremental records into candidate queue and confirms no approved record changes before #325 review.
• Crash matrix: before/after partition read, extraction, proposal staging, checkpoint commit, repo restart, replay; exactly-once observable proposals despite at-least-once processing, with no missed source events.
• Two independent repositories and two Issue partitions demonstrate no context/record leakage, no checkpoint collision, deterministic scoped resume, and one-partition failure isolation.
• correct/retract/reject survives repeat extraction, backfill, merge/split proposal, index lag and rebuild. Source SHA drift and version applicability downgrade to review-needed rather than falsely current.
• Cost and resource caps exercise abort/pause/resume while preserving existing candidate and Controller state; no raw transcript, hidden reasoning, secret, full diff or GitHub auth in prompts/output/state.
• Domain benchmark includes useful workflow extraction, duplicate lesson merging, conflicting/obsolete claim flags, ephemeral PR state excluded, valid negative debugging lesson, and hallucinated unsupported evidence rejected.
• Preview includes exact source scope, event-seq watermark, affected candidate IDs, operator decision requirements and redacted resource estimates; source missing/inconclusive never silently counts as processed.
• Any LLM step executes inside the approved sandbox/provider with existing locks; no default-network/telemetry/cloud changes and no regression to Controller authority, review certificates or existing single-agent mode.
• Docs distinguish MemCurrent inspiration from AutoForge's stricter evidence and approval boundary; no claim that Databricks benchmarks validate GitHub PR workloads. Real provider trials explicitly opt-in.

Non-goals

Not an automatic privilege-bearing Dreaming Agent; not default raw session logging; not an automatic modification of approved memory.md, AGENTS.md, repository files or GitHub issues; not deployment of Databricks Agent Bricks or Beacon; not a mandatory retrieval engine switch or initial MVP release gate.

References

• Databricks MemCurrent: Designing Persistent Memory for Production Agents
• Databricks Agent memory pipelines — checkpoints and maintenance
• #325 — review-gated Knowledge promotion