[Bug] v0.12.1 Linux: parallel_extract reaches 218% of memory budget before fail_whole; watcher repeatedly retries full reindex

Start with src/pipeline/pass_parallel.c at the extraction backpressure and file sorting locations cited in the issue, then read src/foundation/system_info.c for worker counts and src/watcher/watcher.c for retry behavior. The report does not isolate the allocation or provide a deterministic public reproduction, so first investigate bounded ways to reproduce and measure the overshoot. Done means extraction stays within a safe memory limit and repeated resource failures do not trigger the same uncontrolled rebuild loop.

Version

codebase-memory-mcp 0.12.1

Platform

Linux (x64)

Install channel

GitHub release archive / install.sh / install.ps1

Binary variant

standard

What happened, and what did you expect?

Summary

Codebase Memory MCP v0.12.1 still allows severe memory overshoot during parallel extraction, despite the cooperative over-budget abort introduced in PR #2112.

On Ubuntu Linux, a background watcher-triggered full reindex uses 16 workers and reaches 8,279 MiB RSS against a 3,794 MiB budget (218%) before the index attempt can be terminated.

The same workstation experienced two confirmed global OOM kills on October 19, 2026.

This is a follow-up to my earlier report in issue #2649:

https://github.com/DeusData/codebase-memory-mcp/issues/2649#issuecomment-6268807453

Unlike my previous comment, I have now recovered the index worker logs and identified the failing pipeline phase.

Environment

• Codebase Memory MCP: v0.12.1

• OS: Ubuntu 26.04.1 LTS, Linux x86_64

• Kernel: 7.0.0-38-generic

• CPU: Intel Core i5-12500H, 16 logical CPUs

• Physical memory: approximately 14 GiB usable, 4 GiB swap

• Client: Codex

• Installation: official install.sh

• Existing indexed projects: 35

• Trigger: automatic Git watcher reindex

Relevant configuration:

auto_index          = true
auto_index_limit    = 50000
auto_watch          = true
watcher_enabled     = true
index_max_files     = off
index_max_source_mb = off

Actual behavior

1. Automatic indexing triggers a full rebuild

The watcher detects Git changes and launches an index worker.

Worker log:

level=info msg=mem.init budget_mb=3794 total_ram_mb=15192 source=daemon_worker_cap
level=info msg=pipeline.discover files=54424 elapsed_ms=4569
level=info msg=pipeline.route path=format_change_reindex stored_format=0
level=info msg=pipeline.route path=full
level=info msg=parallel.extract.start files=54424 workers=16

The repository contains many archived development and test-evidence directories with nested copies of source files. These contribute to the large discovery count.

2. RSS exceeds the configured budget before aborting

The same worker reports:

level=info msg=parallel.extract.progress done=20 total=54424
level=warn msg=mem.pressure.warn rss_mb=8279 budget_mb=3794 pct=218
level=warn msg=mem.backpressure.futile action=confirm
level=error msg=mem.budget.exceeded rss_mb=6891 budget_mb=3794 phase=parallel_extract action=fail_whole

Only approximately 20 files had completed when the pressure warning appeared.

The worker was already at 218% of its stated budget. The lower RSS in the subsequent mem.budget.exceeded line suggests that some memory may have been reclaimed before the final abort, but the excessive allocation had already occurred.

Three other recovered worker logs exhibit the same pattern:

Discovered files | Budget (MiB) | Highest logged RSS (MiB) -- | -- | -- 54,412 | 3,794 | 5,312 54,424 | 3,794 | 8,279 54,424 | 3,798 | 4,894 54,425 | 3,798 | 5,519

The values represent the highest RSS measurements visible in the extracted log excerpts, not necessarily each process's absolute peak.

All four workers reached parallel_extract and logged mem.budget.exceeded ... action=fail_whole.

3. The Linux kernel records global OOM kills

On October 19, 2026:

09:41:15 — PID 49762
task=codebase-memory
global_oom
anon-rss:7839532kB
10:27:40 — PID 92769

task=codebase-memory

global_oom

anon-rss:7878440kB

Both events were system-wide OOM conditions, rather than container-specific memory-limit failures.

The recovered worker logs are from the same incident window. The exact PID-to-worker-log correspondence has not yet been independently established.

4. The watcher repeatedly retries the failing work

The daemon logs show repeated attempts against the same Git project:

watcher.changed ... strategy=git

daemon.index.worker_budget ... active_jobs=1 aggregate_mb=3798 slice_mb=3798

watcher.index.err ... rc=-1 consecutive=1

...

watcher.index.err ... rc=-1 consecutive=10

Later attempts also report:

index.supervisor.worker_failed outcome=killed exit_code=-1

The watcher does apply retry backoff, but the unsuccessful work is scheduled again. This can repeat the same high-memory rebuild without resolving the underlying problem.

Expected behavior

1. Index workers should be prevented from exhausting host memory even when the extraction phase has large in-flight allocations.

2. A reported memory budget should provide a sufficiently effective bound or be accompanied by an OS-supervised hard safety ceiling.

3. Automatic indexing should fail safely and preserve the existing usable index when its resource requirements exceed supported limits.

4. Repeated resource-exhaustion failures should trigger stronger retry suppression or a circuit breaker.

5. Rebuilds triggered by a format mismatch should expose enough diagnostic information to determine why the existing index cannot be reused.

Source-code analysis

The following implementation details appear relevant:

src/foundation/system_info.c

cbm_default_worker_count(true) uses all available CPU cores unless overridden by CBM_WORKERS.

https://github.com/DeusData/codebase-memory-mcp/blob/v0.12.1/src/foundation/system_info.c#L318-L341

src/pipeline/pass_parallel.c

The extraction backpressure is cooperative and evaluated as workers pull files. Already-running parses can retain substantial memory before the abort is propagated.

https://github.com/DeusData/codebase-memory-mcp/blob/v0.12.1/src/pipeline/pass_parallel.c#L1145-L1240

The extraction phase also sorts files by descending size before processing them, which may amplify concurrent in-flight allocation peaks:

https://github.com/DeusData/codebase-memory-mcp/blob/v0.12.1/src/pipeline/pass_parallel.c#L1591-L1604

src/watcher/watcher.c

Unsuccessful watcher-triggered indexes are retried using backoff. The failed attempt does not permanently disable indexing for that project.

https://github.com/DeusData/codebase-memory-mcp/blob/v0.12.1/src/watcher/watcher.c#L1650-L1710

src/pipeline/pipeline.c

An index format mismatch routes the operation to a full rebuild. The recovered logs repeatedly show stored_format=0.

https://github.com/DeusData/codebase-memory-mcp/blob/v0.12.1/src/pipeline/pipeline.c#L2845-L2861

These observations establish the execution path and memory overshoot. They do not yet identify the individual allocation responsible or prove a persistent memory leak.

Related work

• #2649 — Previous OOM report and my v0.12.1 confirmation.

• #1864 — Historical recurring OOM and full-reindex loop.

• #1973 — Over-budget extraction and OOM instead of graceful degradation.

• #2533 — Repeated watcher-triggered indexing.

• PR #2112 — Merged: cooperative fail_whole memory-budget abort. The recovered logs show that this mechanism activates, but too late to prevent a large RSS overshoot.

• PR #2202 — Merged: memory management and extraction-result spill improvements.

• PR #1724 — Open: supervised worker RSS watchdogs. This appears directly relevant as an independent safety mechanism.

• PR #2082 — Open: bounded two-pass extraction.

• PR #1925 — Open: memory-aware concurrent indexing.

Reproduction status

The observed failure sequence is:

1. Use Codebase Memory v0.12.1 with automatic indexing and watching enabled.

2. Open an existing indexed Git project requiring a full rebuild.

3. Allow the watcher to initiate the rebuild.

4. Observe extraction memory and repeated worker failures.

This has occurred repeatedly on my workstation, but it is not yet a deterministic public reproduction. The exact triggering files and an equivalent public repository have not been isolated.

No proprietary source code is included in this report.

I have not deliberately triggered another OOM because doing so risks destabilizing the workstation.

Diagnostics

CBM_DIAGNOSTICS=1 was not enabled before these incidents, so no diagnostics trajectory is available.

However, the recovered worker logs provide memory budget, measured RSS, pipeline phase, worker count, discovered file count, and abort reason. Linux kernel logs independently confirm the global OOM events.

Request to maintainers

Could you review whether the pending supervised worker watchdog in PR #1724 would address this specific Linux failure mode?

It would also be useful to investigate why parallel_extract reaches 218% of its budget after so few completed files, and whether watcher retries should be suppressed after repeated memory-budget failures.

I can provide additional sanitized logs and test a resource-constrained fix, but would prefer not to reproduce another uncontained global OOM.

Reproduction

Environment

• Codebase Memory MCP v0.12.1
• Ubuntu 26.04.1 LTS, Linux x86_64
• Intel Core i5-12500H (16 logical CPUs)
• Approximately 14 GiB usable RAM and 4 GiB swap
• Codex MCP integration

Steps observed

1. Upgrade Codebase Memory MCP from v0.10.8 to v0.12.1 using the official installer.

2. Enable automatic indexing and Git watching:

   codebase-memory-mcp config set auto_index true
   codebase-memory-mcp config set auto_watch true
   codebase-memory-mcp config set watcher_enabled true
   codebase-memory-mcp daemon stop
   

3. Restart Codex and continue working in a previously indexed Git repository.

4. The Git watcher automatically detects changes and starts a full reindex. The failing repository contains approximately 54,424 discoverable files, including archived development evidence with nested source-code copies.

5. The worker logs show the following sequence:

   pipeline.discover files=54424
   pipeline.route path=format_change_reindex stored_format=0
   pipeline.route path=full
   parallel.extract.start files=54424 workers=16
   

6. Within the first approximately 20 completed files, memory consumption exceeds the worker's budget.

Actual result

The worker reports 8,279 MiB RSS against a 3,794 MiB budget (218%).

The cooperative memory protection subsequently logs mem.backpressure.futile and mem.budget.exceeded ... action=fail_whole, but the memory overshoot has already occurred.

Multiple watcher-triggered attempts fail, and the Linux kernel independently recorded two global OOM kills involving codebase-memory on October 19, 2026.

Expected result

Memory usage should remain safely bounded, or the supervisor should terminate the index worker before it can exhaust host memory.

Repeated memory-budget failures should not automatically trigger another identical high-risk rebuild without effective resource containment.

Reproduction limitations

This is an observed and recurring failure in my environment, not yet a deterministic minimal public reproduction.

The exact triggering source files, repository commit, and equivalent public OSS corpus have not been isolated. The index was initiated automatically by the watcher, not through a manually captured MCP JSON tool call.

No proprietary source code is included.

Previous report

This is a follow-up to my earlier confirmation that the issue persists on v0.12.1:

https://github.com/DeusData/codebase-memory-mcp/issues/2649#issuecomment-6268807453

Logs

=== Worker 1: .worker-log-VeO7h1 ===

level=info msg=mem.init budget_mb=3794 total_ram_mb=15192 source=daemon_worker_cap
level=info msg=pipeline.discover files=54412 elapsed_ms=4853
level=info msg=pipeline.route path=format_change_reindex stored_format=0
level=info msg=pipeline.route path=full
level=info msg=mem.phase phase=structure rss_mb=157 footprint_mb=157 commit_mb=177 tracked_mb=81 peak_mb=158 peak_charged_mb=157
level=info msg=parallel.extract.start files=54412 workers=16
level=warn msg=mem.pressure.warn rss_mb=4119 budget_mb=3794 pct=108
level=warn msg=mem.backpressure.futile action=confirm
level=error msg=mem.budget.exceeded rss_mb=5312 budget_mb=3794 phase=parallel_extract action=fail_whole

=== Worker 2: .worker-log-suz0A5 ===

level=info msg=mem.init budget_mb=3794 total_ram_mb=15192 source=daemon_worker_cap
level=info msg=pipeline.discover files=54424 elapsed_ms=4569
level=info msg=pipeline.route path=format_change_reindex stored_format=0
level=info msg=pipeline.route path=full
level=info msg=mem.phase phase=structure rss_mb=156 footprint_mb=156 commit_mb=182 tracked_mb=81 peak_mb=159 peak_charged_mb=156
level=info msg=parallel.extract.start files=54424 workers=16
level=info msg=parallel.extract.progress done=20 total=54424
level=warn msg=mem.pressure.warn rss_mb=8279 budget_mb=3794 pct=218
level=warn msg=mem.backpressure.futile action=confirm
level=error msg=mem.budget.exceeded rss_mb=6891 budget_mb=3794 phase=parallel_extract action=fail_whole

=== Worker 3: .worker-log-opchiV ===

level=info msg=mem.init budget_mb=3798 total_ram_mb=15192 source=ram_fraction
level=info msg=pipeline.discover files=54424 elapsed_ms=4186
level=info msg=pipeline.route path=format_change_reindex stored_format=0
level=info msg=pipeline.route path=full
level=info msg=parallel.extract.start files=54424 workers=16
level=warn msg=mem.pressure.warn rss_mb=3899 budget_mb=3798 pct=102
level=warn msg=mem.backpressure.futile action=confirm
level=error msg=mem.budget.exceeded rss_mb=4894 budget_mb=3798 phase=parallel_extract action=fail_whole

=== Worker 4: .worker-log-MxhBjC ===

level=info msg=mem.init budget_mb=3798 total_ram_mb=15192 source=ram_fraction
level=info msg=pipeline.discover files=54425 elapsed_ms=4602
level=info msg=pipeline.route path=format_change_reindex stored_format=0
level=info msg=pipeline.route path=full
level=info msg=parallel.extract.start files=54425 workers=16
level=warn msg=mem.pressure.warn rss_mb=5519 budget_mb=3798 pct=145
level=warn msg=mem.backpressure.futile action=confirm
level=error msg=mem.budget.exceeded rss_mb=5303 budget_mb=3798 phase=parallel_extract action=fail_whole

=== Linux kernel OOM events — October 19, 2026 ===

09:41:15 — Killed process 49762 (codebase-memory), anon-rss:7839532kB, global_oom

10:27:40 — Killed process 92769 (codebase-memory), anon-rss:7878440kB, global_oom

=== Watcher behavior ===

watcher.changed ... strategy=git
daemon.index.worker_budget ... active_jobs=1 aggregate_mb=3798 slice_mb=3798
watcher.index.err ... rc=-1 consecutive=1
...
watcher.index.err ... rc=-1 consecutive=10

The worker logs and kernel OOM records are from the same incident period. Exact correspondence between individual worker log files and the OOM-killed PIDs has not yet been established.

The failure occurs during parallel_extract, before registry_build.

Diagnostics trajectory (memory / performance / leak issues)

CBM_DIAGNOSTICS=1 was not explicitly enabled before the incidents.

I searched /tmp for trajectory.ndjson and cbm-diagnostics-*.ndjson, but no matching files were found.

However, the recovered worker logs provide internal memory measurements across repeated attempts:

- Configured worker budget: 3,794–3,798 MiB
- Highest observed RSS: 8,279 MiB
- Maximum observed budget utilization: 218%
- Pipeline phase: parallel_extract
- Worker threads: 16
- Discovered files: approximately 54,424
- Failed attempts: at least four confirmed by recovered logs
- Error: mem.budget.exceeded, action=fail_whole
- Additional evidence: two Linux kernel global OOM kills

The logs demonstrate that cooperative memory-budget enforcement activates only after substantial RSS overshoot.

I have avoided deliberately triggering another uncontained OOM because it destabilizes the entire workstation.

If needed, I can test a patch under a strict OS-enforced memory limit and provide diagnostics without risking another system-wide failure.

Project scale (if relevant)

35 indexed projects. Failing full reindex: 54,412–54,425 discovered files, 16 extraction workers. Largest existing graph: 70,553 nodes / 369,638 edges / 718 MiB on disk.

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.