Node BUG: Mainnet sync: peak RAM nearly doubled in node 11.1.0 (15.4 -> 36.2 GiB)

Open
  • Is something wrong?
#3,669 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
haskell
Domain
databases, distributed-systems

Research direction

Start with the reported 11.0.1 versus 11.1.0 measurements and the linked ouroboros-consensus#2227 pull request, which identifies LedgerDB garbage collection being gated on snapshotting. Compare the relevant backport status and reproduce or validate the sync memory profile; done means confirming whether the backport removes the observed retention regression.

Written by the indexing model from the issue text.

Description

node upstream bug

Summary

Full mainnet sync from genesis with node 11.1.0 consumed roughly twice the memory of
11.0.1 while getting faster. Peak RSS went from 15.43 GiB to 36.15 GiB (+134%), and
peak retained live data from 10.28 GiB to 25.65 GiB (+150%).

Full report, including per-era timings and graphs:
https://docs.google.com/document/d/e/2PACX-1vTujvaT-knCY4ZTUVqUelUIww8n9LJhbd8ZnBjMsUxAp4fvpfmcVFux-qENSzL057vXPxON5LC8_UMk/pub

What was measured

Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, V2InMemory
LedgerDB backend, ghc-9.6). Memory figures are parsed from the node's own Resources:
trace lines, so GC live bytes is what the collector could not reclaim, as distinct from
heap the RTS happens to be holding.

11.0.1 11.1.0 change
Peak RSS 15.43 GiB 36.15 GiB +134%
Mean RSS 12.95 GiB 18.92 GiB +46%
Peak GC live bytes 10.28 GiB 25.65 GiB +150%
Mean GC live bytes 6.03 GiB 9.06 GiB +50%
Major GCs 1,896 1,331 -30%
Total allocation 336 TB 285 TB -15%
Sync time 133,009 s 116,618 s -12.3%
Mean CPU load 152% 156% +2.6%

Speed was not affected: 11.1.0 finished 4 h 33 min faster and advanced 6.9 M slots
further, with CPU load flat. The regression is purely in memory.

No OOM was recorded: the test host has 134 GB, so a 36 GiB peak is survivable there but
not on the 16-32 GB machines many operators run.

Shape of the regression

The two releases produce distinct heap profiles, and the shape is what points at the
cause:

  • 11.0.1 - monotone staircase. Climbs in steps to a 15.34 GiB plateau at slot 134 M
    and never gives memory back: 21 heap decreases across 130,674 samples, nothing above
    20 GiB.
  • 11.1.0 - volatile sawtooth. Oscillates roughly 19-36 GiB with 204 heap decreases.
    34.7% of samples sit above 20 GiB and 6.0% above 30 GiB across 21 separate excursions.
    The pattern starts early: 11.1.0 is already at 9.16 GiB RSS before slot 20 M, where
    11.0.1 is at 2.24 GiB.

Three details indicate retention rather than allocation churn:

  1. The regression is in GC live bytes, i.e. data the collector could not reclaim, not
    just in heap the RTS was holding spare.
  2. Node 11.1.0 ran fewer major GCs than 11.0.1 (1,331 vs 1,896) despite holding far
    more heap, so the heap was ballooning between collections.
  3. It allocated less in total than 11.0.1 (285 vs 336 TB) while retaining much more,
    so this is not extra garbage being produced.

The run took 386 snapshots at a median interval of 5.0 minutes, and the RSS excursions
recur on a comparable cadence - consistent with ledger states accumulating until each
snapshot completed and only then being collected.

Root cause

ouroboros-consensus#2227,
"Decouple LedgerDB garbage collection from snapshotting", merged 2026-08-27 onto
ouroboros-consensus-4.x-backports:

As the thread delay for snapshotting was blocking the snapshotting thread, we would
accumulate states in the LedgerDB until the snapshot was taken.

The delay now runs in its own thread, so LedgerDB garbage collection is no longer gated
on snapshotting.

Dominant language
Python
Stars
59
Forks
28
Avg merge
13h 57m
Merged PRs (30d)
87

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IntersectMBO/cardano-node-tests

All issues in IntersectMBO/cardano-node-tests

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.