Cache per-blob facts by content address for incremental maps and multi-revision history

Start with pkg/scanner/git_cache_test.go and profile.Profile to understand the existing Git fixture and profiling flow, then review the proposed new pkg/cache/ and tests/history/ entry points. The work is complete when cached and uncached outputs are byte-identical, cache invalidation and integrity tests pass, and history fixtures match uncached per-revision results.

User outcome

• Re-profiling a repository after a git pull only costs the changed files.
• Profiling every release tag of a project is a routine command. Example: "Linux language, line-count and component composition across every v-tag."
• CI on a monorepo pays for the diff, not the tree.

Why

• dircue's per-file facts are pure functions of (content bytes, path-derived inputs such as filename, extension and .gitattributes results, ruleset version, and coverage-affecting settings). In Git sources the content is already addressed by its blob ID, so memoization is both sound and nearly free to key.
• Most blobs are unchanged between neighboring commits (typically well over 95%). Incremental maps and multi-revision history become cheap without any cleverness in the observers.
• A time-lapse of a famous codebase's composition, computed in seconds on a laptop, is a memorable demo. It's also a genuinely useful product feature: architecture evolution, language migration tracking, the growth of generated or vendored code.
• Linguist, scc and BCA have no such mode; each run starts from scratch. This is additive value that doesn't re-implement any of them.

Proposal

• An opt-in cache: --cache DIR, or DIRCUE_CACHE_DIR. It stays off by default so a plain run keeps zero writes outside stdout and stderr.
  • The store is content-addressed. Its key is a SHA-256 of: the blob ID (or directory content hash, see #101); the path-dependent inputs actually consulted; the ruleset fingerprint (Linguist data version, Enry and scc and BCA versions, dircue fact-schema version); and the coverage-affecting settings fingerprint (#91).
  • Each value holds the ledger row(s) for that file (#97), including function rows.
  • Every entry is self-verifying: it stores its key inputs and checksum, so a corrupt or foreign entry is a miss, never a wrong answer.
• dircue history (name to be decided). Arguments: --revs A..B, --tags GLOB or --first-parent --every N.
  • It emits one map summary or ledger aggregate per revision, as JSONL, plus a composition time-series table.
  • Blobs are read in pack order and shared across revisions (see #105).
  • Bounds: a maximum number of revisions, and budgets that report partial when hit.
• Multi-file facts (components, edges, deployables) are recomputed from the cached per-file facts and the declaration-file contents. Only the per-file extraction stages are cached, not whole-map aggregation. This keeps correctness simple.
• Cache management: dircue cache stats|prune --max-bytes, with deterministic eviction (LRU by access time is the only nondeterminism, and it never affects output).

Tasks

• Specify the cache key precisely, per observer: which path-derived and attribute inputs each fact consults. Add an audit test that fails if an observer reads an input the key doesn't include.
• Implement the store: a directory of sharded files or a single append-only pack, with atomic writes, a size cap and integrity checks.
• Wire the cache into the ledger row builder, and prove output identity with and without the cache.
• dircue history: revision selection, pack-order blob scheduling, per-revision aggregation, JSONL and table output.
• Documentation: cache semantics, safety (the cache is user-local data, never shared by default) and an invalidation guide.

Test plan

1. Unit tests

New package pkg/cache/ (new); test file pkg/cache/cache_test.go (new).

• Key completeness audit: for each observer (language classifier, scc counter, format inspector, role assigner, BCA summarizer), assert that every input the observer reads is represented in the cache key. Implement as a compile-time or init-time table in pkg/cache/ that enumerates key components; a test verifies the table is non-empty and each component produces a different key when mutated. This is the mandatory audit described in the issue.
• Cold / warm / no-cache identity: given a synthetic multi-file Git fixture, assert that profile.Profile is byte-identical across three runs: --cache DIR (cold), --cache DIR (warm, same dir), and no --cache flag. Use the existing gitFixture helper in pkg/scanner/git_cache_test.go as a model.
• Invalidation mutation tests: for each key component below, flip it and assert the cache produces a miss (not a wrong hit):
  • blob ID (replace content, keep path);
  • path (rename file, keep content);
  • .gitattributes override (add a linguist-language attribute);
  • ruleset fingerprint (bump pkg/codemetrics:EngineVersion);
  • coverage-affecting setting (change MaxFileBytes).
    Five sub-tests, all table-driven.
• Integrity: write a valid cache entry, then corrupt it (flip one byte in the middle). Assert the corrupted entry is treated as a miss and a fresh value is computed and stored.
• Concurrent writers: launch 4 goroutines writing the same key simultaneously on a temp directory. Assert no file corruption, no panic, and the final stored value is one of the valid writes (atomic-write semantics).
• Read-only cache directory: set the cache dir to a directory with mode 0444. Assert that --cache falls back to a no-cache run gracefully (non-fatal error, output unaffected).
• Full disk simulation: wrap the store's write path with an injected error that returns ENOSPC. Assert graceful degradation, no partial entries left on disk.
• Foreign version entry: place a cache entry written by a future hypothetical version (bumped schema version field). Assert it is a miss, not a crash.

2. Independent oracles

Harness: tests/history/ (new).

• The independent oracle for history correctness is a sequence of single-revision dircue map runs (one per tag), each compared against the corresponding entry in the history output. No external tool is needed here; the oracle is dircue itself in uncached single-revision mode.
• For the cache store itself: run sha256sum on every stored entry after a warm run. Assert checksums match the re-computed key hash (self-verifying property). This requires no external tool.
• Tie to #84 by noting that the conformance oracle (dircue-linguist:9.7.0) already gates the underlying language facts; the cache layer does not alter them.

3. Hand-labeled ground truth and fixtures

• Commit 5 small synthetic Git fixtures under tests/history/fixtures/ (new). Each fixture has 3–5 commits, a known composition change between commits, and a hand-verified .expected-history.jsonl (committed as plain text; total size well under 100 KiB).
• For the corpus history run: the harness fetches a bare kernel clone via tests/performance/fetch.py (existing) on demand; never committed. Per #88, only the receipt and expected-aggregate files are committed, not the corpus.
• Precision gate: the warm-cache aggregate for every fixture revision must equal the cold-run aggregate exactly (bit-for-bit identity); there is no tolerance.

4. Metamorphic invariants (#85)

• Cold = warm = no-cache: output is byte-identical across all three cache modes for every corpus fixture and source mode. Assert in pkg/cache/cache_test.go.
• Worker-count invariant: cache hit rate and output are identical with --workers 1 and --workers 8 for the same warm cache. Assert in pkg/cache/cache_test.go.
• Revision independence: profiling revision A after revision B (warm cache) gives the same output as profiling revision A from scratch (cold). Assert in tests/history/e2e.py.
• Shallow clone: running dircue history on a shallow clone that lacks some commits emits coverage: partial, reason: history_unavailable for the missing range, not a fabricated aggregate. Assert in a dedicated tests/history/fixtures/shallow/ fixture (new).

5. Fuzz targets (#87)

New file pkg/cache/fuzz_test.go (new), on-demand only (never campaigns in CI per #87):

• FuzzCacheKeyRoundtrip(f *testing.F): seeds with known key inputs (blob ID, path, settings hash); checks that MarshalKey(UnmarshalKey(k)) == k and that distinct inputs produce distinct keys.
• FuzzCacheEntryDecode(f *testing.F): seeds with valid serialized entries; checks that a mutated byte either decodes cleanly as a different value or is rejected as corrupt — never panics.

6. Mutation-testing focus (#86)

Priority mutation targets:

• The keyIncludes(blobID) check in the key builder — a mutant omitting the blob ID must be killed by the blob-swap invalidation test.
• The integrity checksum comparison in the cache reader — a mutant that skips the checksum check must be killed by the corruption test.
• The atomic write path (write to temp file, then rename) — a mutant that writes directly must be killed by the concurrent-writer test.

Use go-mutesting on pkg/cache/ only.

7. Performance and resource checks (#89/#90)

• Cache hit rate counter (PR CI): expose a CacheHits and CacheMisses counter in pkg/cache/. In the warm-cache test, assert CacheHits / (CacheHits + CacheMisses) >= 0.95 on a fixture where 95%+ of blobs are unchanged across two adjacent revisions. This counter is deterministic and belongs in the test job.
• Warm-cache speedup (on-demand): in tests/history/e2e.py, assert that warm-cache re-profiling of a large corpus revision completes within 10% of the cold-run time per the issue's success criterion. Log speedup ratio to JSONL. Run via make history-perf (new target) or workflow_dispatch only.
• Disk-use cap: assert that dircue cache prune --max-bytes N leaves the cache dir under N bytes with correct eviction ordering. Unit test in pkg/cache/cache_test.go.

8. E2E script

tests/history/e2e.py (new). JSONL log fields per revision:

{"revision": "v6.1", "wall_s": 1.2, "cache_mode": "cold|warm",
 "cache_hits": 91000, "cache_misses": 120, "bytes_inflated": 45000000,
 "rss_mib": 220, "languages_top3": [...], "mismatch_vs_uncached": false}

Steps:

1. Fetch bare kernel clone via tests/performance/fetch.py; list all v* tags up to a configurable --max-revisions cap.
2. Run dircue history --tags 'v*' --cache $TMP (cold); log per-revision stats.
3. Run dircue history --tags 'v*' --cache $TMP (warm); log per-revision stats and hit rates.
4. For each revision, run an uncached single-revision dircue map; compare aggregates against the history output. Assert exact match.
5. Emit a summary table: revision, cold wall time, warm wall time, speedup, hit rate.
6. Exit non-zero if any mismatch_vs_uncached == true or if warm speedup < 3× on a corpus where > 90% of blobs are unchanged.

Invoke: make history-e2e (new target) or workflow_dispatch. Never schedule:.

9. CI placement

What	Where	Trigger
Unit tests (pkg/cache/...), cold/warm/no-cache identity, invalidation, integrity	existing test job in ci.yml	every PR and push
Cache hit-rate counter gate (fast, deterministic)	existing test job	every PR and push
History correctness on committed fixtures (tests/history/fixtures/)	new step in test job	PRs touching pkg/cache/ or pkg/scanner/
History E2E + speedup benchmark (e2e.py)	workflow_dispatch + make history-e2e	on-demand only
Disk-use cap and prune correctness	existing test job	every PR and push

No schedule: trigger anywhere.

10. Regression fences

• Key-completeness regression: the audit table in pkg/cache/ is a compile-time list; if an observer gains a new input that affects output, the table must be updated or a test fails. Add a TestKeyCompletenessTableNonEmpty test as a hard guard.
• Shallow-clone contract: tests/history/fixtures/shallow/ (new, committed) is a permanent regression fixture for the "never fabricate history" contract. Any future refactor of the history walker must keep this test passing.
• Output-identity regression: the existing parity receipts (tests/conformance/results/latest.json, tests/performance/kernel_git/receipt.json) are the gates. A cache implementation that silently alters output would fail these receipts; they need not be duplicated.

Risks

• Key completeness. Key completeness is the whole game; an incomplete key causes wrong answers. The audit test is mandatory.
• Disk use. Cache disk use on huge histories needs caps.
• Shallow clones. Their history is unavailable; report unknown, never fabricate.

Success criteria

• Warm-cache re-profiling after a typical pull takes under 10% of the cold time on large repos.
• Kernel history over all major tags completes within a published budget.
• Output is identical with and without the cache.

Related: #10, #39, #89, #90, #27 (churn-aware hotspots can build on history). Related idea issues: #97, #105, #101.

Sequencing: Depends on #97, because cached values are ledger rows. It benefits from #101 (directory-mode keys) and #105 (pack-order reads).

---

This proposal came out of an idea-wizard session on 2026-09-23. It isn't a release commitment, and the milestone is deliberately unset so the owner can decide per idea. It must preserve dircue's contracts:

• deterministic, offline output, with no execution of inspected content;
• the legacy Linguist-compatible CLI and output unchanged;
• unknown never rendered as absent;
• no provider findings or verdicts imported;
• CI event-driven only (no schedule:).