epic: make GitForge CI execution resource-efficient, incremental, and storage-deduplicated

Start with the linked workstream issues #338–#341 and the existing scheduler, lease, cache-mount, storage-trait, and smart-CI graph code, while restoring the Gate 0 correctness failures on current main. Implement one independently deployable phase with reproducible before/after benchmarks and exact-head GitHub/GitForge validation. Done means the phase meets its stated invariants, rollback instructions, recorded benchmark artifacts, and required checks without weakening full validation.

Objective

Make GitForge reliably serve many repositories and concurrent coding-agent/Actions workloads without queue collapse, avoidable rebuilds, repeated downloads, excessive I/O, or unsafe cleanup.

This is an implementation epic, not a proposal to replace GitForge's existing scheduler or storage stack wholesale. Reuse the current durable scheduler, leases, cache mounts, job receipts, storage traits, and existing smart-CI graph work where they pass review. Measure first; ship in independently deployable phases.

Why this is needed

The current main branch already has repository-aware fairness and a special resource-class guard for workspace Cargo tests, as well as cache mounts and durable job scheduling. It still lacks a complete multi-dimensional host-pressure admission model, separate fast/heavy capacity, a content-addressed immutable artifact store, managed copy-on-write workspaces, and sound per-test result reuse.

Evidence captured during the October 31 audit:

• The October 10 worktree census recorded 44 live worktrees, including 17 dirty WIP trees and one 3,962-file build-artifact spill.
• The current cache store is key-addressed file storage, not a shared content-addressed object store. The CI cache mount is keyed by image tag.
• The current queue's priority values are Normal/High/Low and it has repository-aware running-job fairness. Resource classification currently includes a special case for workspace-wide Cargo test jobs rather than a general CPU/memory/I/O/scratch reservation contract.
• Existing issue #277 records severe host pressure, D-state waits, heavy disk I/O and low free disk during concurrent CI activity. Issue #284 records overlapping Actions/GitForge validation lanes competing for the same self-hosted pool.
• The prior smart-CI branch has useful graph implementation work, but it is six commits ahead and 70 commits behind current main. Rebase and review it; do not merge it unchanged.
• Main at commit 5ceb106d had failed validation: an SSH callback signature compile error, an API test fixture compile error, 15 unvetted cargo-vet dependencies, and GitForge CI polling that returned HTTP 401. PR #336's storage tests/build/clippy/coverage were green at its inspected head, while Supply Chain and GitForge CI polling remained failing. These correctness/integration gates are blockers, not work that storage optimization can bypass.

Workstreams

1. Resource-aware queues and admission — #338. Separate control-plane responsiveness from compute; provide fast, standard, heavy and background lanes; weighted fair scheduling across tenants/repos; per-repo concurrency; reserved fast capacity and anti-starvation; resource reservations/enforcement; host-pressure-based deferral with durable reason codes.
2. Shared storage and managed workspaces — #339. Add immutable CAS for non-Git artifacts, manifests/reference tracking, safe cross-process publication, trust namespaces, worktree lifecycle management, optional OverlayFS/reflink/Btrfs providers, two-phase GC and storage telemetry. Keep Git's object database canonical for Git objects.
3. Incremental test impact and reusable results — #340. Rebase/review the existing graph branch; map tests to executed code, select the safe impacted set, use complete test fingerprints, cache only deterministic eligible outcomes, and fall back to broader validation when graph confidence is inadequate. Integrate the OpenTestSuite prototype via an adapter contract.
4. Toolchain and database benchmark bake-off — #341. Evaluate cargo-nextest, sccache, cargo-chef, mold/LLD, lust-lang, ts-rust/tsc-rs, Biome/Oxlint, uv/Ruff/pytest-xdist/testmon and database tuning against exact correctness corpora and representative resource profiles.

Required execution order

Gate 0 — restore correctness and safe execution

• Resolve the current main compile/test failures and the GitForge CI credential/polling contract.
• Resolve cargo-vet attestations by auditing dependencies; do not blanket-certify or suppress.
• Complete public fork runner isolation and required-check semantics (#318).
• Validate the open atomic storage publication PR (#336) at exact head; merge only after all required GitHub and self-hosted checks have valid evidence.
• Preserve the separate SSH push trigger contract work (#258).

Gate 1 — baseline observability and admission

• Capture queue wait, lane, queue reason, runner capacity, CPU/memory/I/O/disk pressure, per-job resource use, cache hit/miss, workspace creation time and bytes read/written.
• Add bounded admission and fair fast/heavy capacity before turning up parallelism.
• Keep active jobs alive when headroom becomes unsafe; defer new work.

Gate 2 — reuse and isolation

• Implement CAS objects/manifests, isolated workspace providers and safe GC behind opt-in flags.
• Benchmark native Git worktree, reflink/OverlayFS and Btrfs snapshot methods on supported filesystems.
• Never share mutable target directories or writable cache objects across unrelated jobs.

Gate 3 — sound incremental validation

• Rebase the prior graph work and validate it against current manifests and pipeline contracts.
• Pilot per-test dependency capture and selected-vs-full differential execution.
• Keep full validation on scheduled/release gates; no heuristic may turn a required check green.

Gate 4 — toolchain and rollout

• Benchmark alternatives by correctness first, then cold/warm latency, CPU-seconds, RSS, disk I/O and cost.
• Roll out per runner profile and per project, with automatic fallback and operator kill switches.

Non-negotiable invariants

• Durable queue, job lease, receipt and restart-recovery semantics stay authoritative.
• New/unknown tests and failed/flaky tests are not skipped.
• Graph/index/cache errors fail open to a conservative full validation scope.
• No untrusted job can write to or poison a shared cache/store used by trusted work.
• Worktree/GC operations require durable ownership/lease checks and never delete dirty live work.
• Snapshots are not backups; retain independent backup/restore procedures.
• Raw logs/conversations remain recoverable; compacted knowledge must carry hashes and source offsets. A 2% size target is measured, not guaranteed.
• No performance claim is accepted without reproducible before/after evidence.

Existing related issues

• #277 resource/storage headroom
• #284 duplicate validation lanes and runner concurrency
• #318 public PR runner trust
• #322 smart CI and dependency/change-impact graph
• #288 push response decoupling
• #335 Amortyx-Stack repo registration / release-gate evidence
• #336 atomic file-backed publication
• #337 release-bundle contract test fix
• #258 successful SSH push CI trigger

Acceptance criteria

• Under mixed-project load, fast checks and control-plane operations remain responsive while heavy jobs are deferred fairly.
• Queue growth is bounded and every deferred job has a measurable reason.
• Build/dependency/test reuse is measured by digest and avoids unsafe mutable sharing.
• Worktree, cache and object GC are crash-safe and cannot delete live work.
• Incremental test selection has zero missed failures in differential validation and maintains full scheduled/release suites.
• GitForge reports p50/p95/p99 API latency, queue wait, throughput, resource utilization, cache efficiency, physical/logical storage, and correctness fallback rate.
• All changes are covered by exact-head GitHub and GitForge validation, rollout/rollback instructions and recorded benchmark artifacts.

The implementation should land as small, reviewable PRs under these workstream issues. Avoid a monolithic rewrite.