Diff document rebuilds for a large untracked (non-.gitignore'd) directory cause multi-GB memory growth on every workspace refresh

Reproduce with a repository containing 100,000+ untracked files and inspect the diff_document.paged_build, diff_document_build_end, and diff_document_manifest_service logs. Trace the manifest rebuild and residency paths to determine why superseded documents remain resident, then add coverage showing memory or document residency stays bounded across refreshes.

Mood: 😊
Category: Bugs

Summary

When a workspace contains a large untracked directory (100k+ files) that isn't excluded via .gitignore, the app repeatedly rebuilds a full diff-document manifest across all of those files on routine workspace-change refreshes. Each rebuild materializes tens of millions of row/anchor entries and appears to remain resident rather than being evicted, causing the main app process to grow by several GB per rebuild — reaching 20+ GB of physical footprint within a couple of hours of normal use.

Environment

What happened?

The app's own internal logs surfaced the mechanism directly. It repeatedly logs a warning:

Skipping untracked additions count for runaway directory (likely missing .gitignore entry); files will show +0 top_dir=<dir> file_count=110000+ cap=1000

Despite this "skip" warning suggesting the directory is excluded from the file count, the diff engine still performs full manifest builds over the same file set whenever get_diff_document_manifest runs (triggered by opening a session, sending a prompt, or periodic workspace-change subscriptions). Log evidence from a single session:

diff_document.paged_build{files=111506} ... rows=81468112 anchors=111506 line_anchors=81248952 chunks=203671 manifest_only=true
diff_document_build_end ... total_rows=81468112 files=111506 duration_ms=1666
diff_document.residency_changed document_id="...:uncommitted:<hash>" documentResidentChunks=1
diff_document_manifest_service ... manifest_service_ms=73171 ... enrich_ms=64052 build_ms=1666

This full rebuild (producing ~81 million row/anchor entries, taking 70-95 seconds of internal processing) recurred at least 5 times within about 7 minutes of normal interactive use (opening sessions, sending prompts), each producing a new document_id and each logged as documentResidentChunks=1 (resident in memory), without an accompanying log entry indicating the previous document was evicted.

Correlated with this, the main app process's physical footprint (via vmmap -summary) grew from a healthy baseline of ~200-500 MB to over 20 GB, then continued oscillating between roughly 5 GB and 25 GB depending on activity, driving system swap to its limit and heavy memory-compressor activity. leaks on the process showed only ~220 MB of classically unreachable memory, indicating this is not a simple pointer leak but unbounded retention/caching of legitimately-referenced diff data.

Steps to reproduce

1. Open a local repository whose working tree contains a directory with 100,000+ files that is not excluded via .gitignore (e.g., a data/ or similar bulk-file directory).
2. Open a session/workspace against that repository in the app.
3. Interact normally — send prompts, switch sessions, let workspace-change subscriptions fire.
4. Watch the app's log file (~/.copilot/logs/github-app.<pid>.log) for repeated diff_document.paged_build{files=...} / diff_document_build_end entries with very large total_rows/anchors values.
5. Monitor the main app process's physical footprint via Activity Monitor or vmmap -summary <pid> — observe multi-GB jumps correlating with each rebuild.

Expected behavior

• A directory that's excluded from the untracked-file count for being a "runaway directory" should also be excluded (or at least capped/paginated) from full diff-manifest builds, not just from the display count.
• Diff documents superseded by a newer rebuild for the same workspace/changes-mode should be evicted from memory promptly instead of accumulating.
• Memory usage should stay roughly bounded regardless of working-tree size, especially for directories the app has already flagged as too large to process normally.

Additional context

• vmmap -summary MALLOC_SMALL zone alone showed the process reaching ~15+ GB across millions of small allocations, consistent with many small per-row/per-anchor diff objects.
• This reproduces reliably and quickly (within minutes) given a sufficiently large untracked directory — happy to provide further sanitized vmmap/log excerpts if helpful for triage.
• Workaround in the meantime: manually excluding the large directory via .gitignore prevents the repeated full-manifest rebuilds and stops further growth.