Log file gets flooded

Reproduce the issue by compiling and running a fresh Testnet node on CentOS Stream 8, then trace the logging around pinned block cache eviction and forest import errors. Done means syncing no longer floods the log with repeated messages that grow it beyond 1 GB.

Did you read the documentation and guides?

• I have inspected the documentation.

Is there an existing issue?

• I have searched the existing issues.

Description of the problem

When syncing from scratch the log file gets flooded with messages about the pinned block cache limit and forest errors. Within 8 hours the log grew >1 GB.

Parts of the log when booting with fresh database:

2023-10-26 21:58:28 Aleph Node
2023-10-26 21:58:28 ✌️ version 0.12.0-1a00a8fd94e
2023-10-26 21:58:28 ❤️ by Cardinal:Aleph Zero Foundation, 2021-2023
2023-10-26 21:58:28 📋 Chain specification: Aleph Zero Testnet
2023-10-26 21:58:28 🏷 Node name: StakeSafe Amsterdam
2023-10-26 21:58:28 👤 Role: AUTHORITY
2023-10-26 21:58:28 💾 Database: RocksDb at /home/adminbart/aleph-node/data/chains/testnet/db/full
No backup path provided, using default path: "/home/adminbart/aleph-node/data/backup-stash" for AlephBFT backups. Please do not remove this folder
2023-10-26 21:58:33 Using default protocol ID "sup" because none is configured in the chain specs
2023-10-26 21:58:33 🏷 Local node identity is: 12D3KooWMVDLapz4BLSfy2cAS7Z4QEi3k4Jo6y4FSwskLGyQffUw
2023-10-26 21:58:33 🔍 Discovered new external address for our node: /ip4/x.x.x.x/tcp/30333/p2p/12D3KooWMVDLapz4BLSfy2cAS7Z4QEi3k4Jo6y4FSwskLGyQffUw
2023-10-26 21:58:33 💻 Operating system: linux
2023-10-26 21:58:33 💻 CPU architecture: x86_64
2023-10-26 21:58:33 💻 Target environment: gnu
2023-10-26 21:58:33 💻 CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
2023-10-26 21:58:33 💻 CPU cores: 2
2023-10-26 21:58:33 💻 Memory: 15384MB
2023-10-26 21:58:33 💻 Kernel: 4.18.0-448.el8.x86_64
2023-10-26 21:58:33 💻 Linux distribution: CentOS Stream 8
2023-10-26 21:58:33 💻 Virtual machine: yes
2023-10-26 21:58:33 📦 Highest known block at #39451231
2023-10-26 21:58:33 〽 Prometheus exporter started at 127.0.0.1:xxxx
2023-10-26 21:58:33 Running JSON-RPC server: addr=0.0.0.0:xxxx, allowed origins=["*"]
2023-10-26 21:58:33 Clique Network status: not maintaining any connections;
2023-10-26 21:58:33 Running session 43834.
2023-10-26 21:58:33 Network status report: authentication connected peers - 0; block sync connected peers - 0;
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xf3b8…0d70
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x9001…0c26
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xc860…1fc9
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x07d6…f34a
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xe51a…536e
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xcd20…2ffb
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x2ab4…f209
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xb51f…5008
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x5315…e12f
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x73ed…d898
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x5864…fa90
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xf8ef…584e
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xf61a…b40d
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xa599…6928
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x3f35…fb35
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x311a…caef
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x245d…4c18
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xa538…26c1
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x50b7…4d1a
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x8fdd…b930
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xddc5…8992
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x51b2…1561
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xccfb…86b4
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xe351…8945
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xbe3a…fbb8
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x3b7e…8d5b
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x67eb…8465
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x1647…ea1f
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xbbe5…d5ea
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x3d12…d39b
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x1796…b26e
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0xbb4e…ed88
2023-10-26 21:58:37 Pinned block cache limit reached. Evicting value. hash = 0x8ee7…34dc
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:10 Error marking block as imported: forest error: block is too new.
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x5901…df5d
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x79ba…fae4
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xeaf3…096f
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x1f9c…3d72
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xab6a…fa97
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x8f93…9f13
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x3ec5…f608
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x3737…001e
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x3bf6…19ab
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xd881…729b
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x345d…115f
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x71c6…65c3
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xe795…fb41
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x9725…91e7
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xf7bf…f22c
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xbca3…ecb4
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x1a02…8134
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x485b…4f7c
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x51b0…3172
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x6967…3353
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xf501…7551
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x04be…c175
2023-10-26 21:59:22 ⚙️ Syncing 554.0 bps, target=#45074098 (8 peers), best: #39460458 (0x0501…b7e5), finalized #39459380 (0x3c98…c687), ⬇ 9.8MiB/s ⬆ 99.1kiB/s
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xed02…e14d
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x214f…1800
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x708e…1407
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x68e8…3daf
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xcb08…99c2
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x28b6…8e22
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x65e1…0bd2
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x25b8…6e0f
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x63b2…9b3a
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xef62…4bfc
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x1f2b…5d83
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xb86c…e296
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x1a22…54f4
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x5a9e…eca8
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xb449…67df
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x24d7…3137
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0x35d7…58c8
2023-10-26 21:59:22 Pinned block cache limit reached. Evicting value. hash = 0xd316…67a5

Information on your setup.

Testnet
Version: 0.12.0-1a00a8fd94e
Running directly on CentOS Stream 8

Steps to reproduce

Compile the code and run a fresh node on testnet

Did you attach relevant logs?

• I have attached logs (if relevant).