Node BUG: Mainnet sync: node 11.1.1 still retains ~55% more live data than 11.0.1

Open
  • Is something wrong?
#3,670 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
haskell
Domain
backend, databases, performance

Research direction

Start with the linked full sync report and compare the 11.0.1 and 11.1.1 runs at revision c2ebdc87dfe07706a83e52f219e712c60d1b0a56, focusing on the divergence from Alonzo onward. Done means determining whether the higher V2InMemory LedgerDB retention is expected or a regression, and identifying a consensus fix or clear evidence for the conclusion.

Written by the indexing model from the issue text.

Description

node upstream bug

Summary

Node 11.1.1 resolved the memory regression reported in #3669, but full mainnet sync
still uses noticeably more memory than the 11.0.1 baseline:

  • Peak RSS +37% (15.43 -> 21.16 GiB)
  • Peak GC live bytes +55% (10.28 -> 15.94 GiB)

This is not the same fault as #3669. That one was an oscillation - heap swinging between
19 and 36 GiB because LedgerDB garbage collection was gated on snapshotting, fixed by
ouroboros-consensus#2227.
Node 11.1.1 no longer oscillates; it sits on a higher plateau than 11.0.1 did, which
suggests a different cause rather than an incomplete fix.

Under investigation. It is not yet clear whether this is expected for the V2 LedgerDB
backend or a genuine second regression, and there is no consensus fix for it yet.

Full report, including per-era timings and graphs:
https://tests.cardano.intersectmbo.org/test_results/sync_reports/mainnet_11_1_1.html

Measurements

Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, V2InMemory
LedgerDB backend, ghc-9.6). Memory is parsed from the node's own Resources: trace
lines, so GC live bytes is what the collector could not reclaim, as distinct from heap
the RTS happens to be holding.

11.0.1 11.1.1 change
Peak RSS 15.43 GiB 21.16 GiB +37%
Mean RSS 12.95 GiB 15.67 GiB +21%
Peak GC live bytes 10.28 GiB 15.94 GiB +55%
Mean GC live bytes 6.03 GiB 6.86 GiB +14%
Major GCs 1,896 1,940 +2%
Total allocation 336 TB 286 TB -15%
Sync time 133,009 s 110,606 s -16.8%
Mean CPU load 152% 157% +3.3%

Node revision under test: c2ebdc87dfe07706a83e52f219e712c60d1b0a56.

Speed is not affected and in fact improved by 6 h 13 min. Allocation went down while
retention went up, so this is not extra garbage being produced. Major GC count is normal,
so the collector is running as often as it did on 11.0.1 - it simply has more it cannot
free.

The heap profile is healthy, just higher

  • 11.0.1 - monotone staircase to a 15.34 GiB plateau reached at slot 134 M. 21 heap
    decreases across 130,674 samples, nothing above 20 GiB.
  • 11.1.1 - also a staircase, plateauing near 19-20 GiB. 90 heap decreases, 3.5% of
    samples above 20 GiB, none above 25 GiB.

For contrast, 11.1.0 had 204 heap decreases with 34.7% of samples above 20 GiB and 6.0%
above 30 GiB. The sawtooth is gone.

It is not chain growth

The 11.1.1 run covered 8.2 M more slots and 2.8 GB more chain than the 11.0.1 run, which
is far too little to explain the gap:

  • The divergence starts around slot 40 M and widens monotonically from there.
  • Truncating all runs to a common window of slots <= 188 M leaves the peaks unchanged:
    15.43 GiB for 11.0.1 against 21.16 GiB for 11.1.1.
  • Node 11.1.1 crosses the 15.34 GiB level that 11.0.1 uses as its ceiling at slot 71 M,
    roughly halfway through Alonzo, with more than half the chain still to sync.

Peak RSS by slot band, GiB:

Slot band 11.0.1 11.1.1
0-10 M 2.24 2.13
20-30 M 4.09 4.84
40-50 M 8.26 9.69
60-70 M 11.49 15.33
80-90 M 13.55 18.12
100-110 M 14.15 19.01
120-130 M 15.43 21.16
150-160 M 15.43 20.36
180-190 M 15.43 20.36

Early sync is at parity - 11.1.1 is actually marginally lower through the Byron era - and
the two curves separate from Alonzo onward.

Why it matters

A 21 GiB peak is comfortable on the 134 GB test host, and no OOM occurred. It is less
comfortable for operators on 16-32 GB machines, which is the constituency the 11.0.1
figure of 15.4 GiB was already close to.

Dominant language
Python
Stars
59
Forks
28
Avg merge
13h 57m
Merged PRs (30d)
87

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IntersectMBO/cardano-node-tests

All issues in IntersectMBO/cardano-node-tests

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.