Node BUG: Mainnet sync: peak RAM nearly doubled in node 11.1.0 (15.4 -> 36.2 GiB)

Aperta
  • Qualcosa non va?
#3,669 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
25/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
haskell
Ambito
databases, distributed-systems

Direzione di ricerca

Inizia dalle misurazioni riportate per 11.0.1 rispetto a 11.1.0 e dalla pull request collegata ouroboros-consensus#2227, che identifica il fatto che la garbage collection di LedgerDB è subordinata allo snapshotting. Confronta lo stato del backport pertinente e riproduci o convalida il profilo di memoria della sincronizzazione; il lavoro è completato quando viene confermato se il backport rimuove la regressione di retention osservata.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

node upstream bug

Summary

Full mainnet sync from genesis with node 11.1.0 consumed roughly twice the memory of
11.0.1 while getting faster. Peak RSS went from 15.43 GiB to 36.15 GiB (+134%), and
peak retained live data from 10.28 GiB to 25.65 GiB (+150%).

Full report, including per-era timings and graphs:
https://docs.google.com/document/d/e/2PACX-1vTujvaT-knCY4ZTUVqUelUIww8n9LJhbd8ZnBjMsUxAp4fvpfmcVFux-qENSzL057vXPxON5LC8_UMk/pub

What was measured

Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, V2InMemory
LedgerDB backend, ghc-9.6). Memory figures are parsed from the node's own Resources:
trace lines, so GC live bytes is what the collector could not reclaim, as distinct from
heap the RTS happens to be holding.

11.0.1 11.1.0 change
Peak RSS 15.43 GiB 36.15 GiB +134%
Mean RSS 12.95 GiB 18.92 GiB +46%
Peak GC live bytes 10.28 GiB 25.65 GiB +150%
Mean GC live bytes 6.03 GiB 9.06 GiB +50%
Major GCs 1,896 1,331 -30%
Total allocation 336 TB 285 TB -15%
Sync time 133,009 s 116,618 s -12.3%
Mean CPU load 152% 156% +2.6%

Speed was not affected: 11.1.0 finished 4 h 33 min faster and advanced 6.9 M slots
further, with CPU load flat. The regression is purely in memory.

No OOM was recorded: the test host has 134 GB, so a 36 GiB peak is survivable there but
not on the 16-32 GB machines many operators run.

Shape of the regression

The two releases produce distinct heap profiles, and the shape is what points at the
cause:

  • 11.0.1 - monotone staircase. Climbs in steps to a 15.34 GiB plateau at slot 134 M
    and never gives memory back: 21 heap decreases across 130,674 samples, nothing above
    20 GiB.
  • 11.1.0 - volatile sawtooth. Oscillates roughly 19-36 GiB with 204 heap decreases.
    34.7% of samples sit above 20 GiB and 6.0% above 30 GiB across 21 separate excursions.
    The pattern starts early: 11.1.0 is already at 9.16 GiB RSS before slot 20 M, where
    11.0.1 is at 2.24 GiB.

Three details indicate retention rather than allocation churn:

  1. The regression is in GC live bytes, i.e. data the collector could not reclaim, not
    just in heap the RTS was holding spare.
  2. Node 11.1.0 ran fewer major GCs than 11.0.1 (1,331 vs 1,896) despite holding far
    more heap, so the heap was ballooning between collections.
  3. It allocated less in total than 11.0.1 (285 vs 336 TB) while retaining much more,
    so this is not extra garbage being produced.

The run took 386 snapshots at a median interval of 5.0 minutes, and the RSS excursions
recur on a comparable cadence - consistent with ledger states accumulating until each
snapshot completed and only then being collected.

Root cause

ouroboros-consensus#2227,
"Decouple LedgerDB garbage collection from snapshotting", merged 2026-08-27 onto
ouroboros-consensus-4.x-backports:

As the thread delay for snapshotting was blocking the snapshotting thread, we would
accumulate states in the LedgerDB until the snapshot was taken.

The delay now runs in its own thread, so LedgerDB garbage collection is no longer gated
on snapshotting.

Lingua principale
Python
Stelle
59
Fork
28
Merge medio
8h 27m
PR unite (30g)
88

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di IntersectMBO/cardano-node-tests

Tutte le issue di IntersectMBO/cardano-node-tests

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.