db-sync stuck in crash loop: deeply nested tx metadata causes `stack depth limit exceeded` on `tx_metadata` insert (Preview)

Open
  • Is something wrong?
#2,187 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
haskell, postgresql
Domain
backend, databases

Research direction

Start by reading src/Cardano/DbSync/Era/Universal/Insert/Grouped.hs around insertBulkTxMetadataPiped, then trace how decoded metadata is rendered and inserted. The issue suggests checking nesting depth before insertion or handling PostgreSQL SQLSTATE 54001; inspect existing tests for metadata insertion and add coverage for deeply nested valid metadata. Done means syncing continues past block 4821137 while retaining the offending metadata bytes.

Written by the indexing model from the issue text.

Description

Summary

db-sync crashes, and keeps crashing after every restart, while inserting the metadata of block 4821137 on Preview. One transaction in that block carries metadata under label 1 that is a list nested 16,053 levels deep ([[[[…[0]…]]]]). The transaction is valid on-chain because the ledger does not limit metadata nesting depth. When PostgreSQL casts the JSON rendering of that metadata with json::jsonb, it exceeds max_stack_depth. The whole block transaction is rolled back, and db-sync can't get past this block.

Because the transaction is valid, every db-sync instance syncing Preview with tx_metadata enabled and default Postgres settings should hit this.

Environment

  • Network: Preview
  • cardano-db-sync / PostgreSQL versions: not captured in the log, will add on request
  • PostgreSQL max_stack_depth = 2048kB (default)
  • Running in a container (podman)

Affected block

Error

[db-sync-node:Info:155] [2026-11-04 12:39:01.56 UTC] Received block which is not in the db with HeaderFields {headerFieldSlot = SlotNo 127055999, headerFieldBlockNo = BlockNo 4821137, headerFieldHash = e5a2152e974d69023be7e7bb8e96141e65d28c2473836285ef06f4575833cb87}. Time to restore consistency.
[db-sync-node:Info:155] [2026-11-04 12:39:01.56 UTC] Starting at epoch 1470
[db-sync-node:Error:155] [2026-11-04 12:39:01.61 UTC] Error SNErrDbSessionErr:
 App call chain: runDbSyncTransaction at Cardano.DbSync.Default:src/Cardano/DbSync/Default.hs:57
 DB call chain: fn:insertBulkTxMetadataPiped md:Cardano.DbSync.Era.Universal.Insert.Grouped loc:src/Cardano/DbSync/Era/Universal/Insert/Grouped.hs:345
 ResultError (ServerError "54001" "stack depth limit exceeded" Nothing (Just "Increase the configuration parameter \"max_stack_depth\" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.") Nothing)
 QueryError "INSERT INTO tx_metadata (key, json, bytes, tx_id)  SELECT key, json::jsonb, bytes, tx_id FROM UNNEST ($1::numeric[], $2::text[], $3::bytea[], $4::bigint[]) AS t(key, json, bytes, tx_id)  RETURNING id" [...]
...
cardano-db-sync: Error SNErrDbSessionErr: (same as above, process exits)

The full query parameters are about 114 KB. The relevant parts:

  • key array: [55534473, 721, 674, 55534473, 55534473, 1, 674]
  • tx_id array: [7083184, 7083185, 7083187, 7083188, 7083190, 7083191, 7083192]
  • The 6th entry (key 1, tx_id 7083191):
    • json: [ × 16,053, then 0, then ] × 16,053 (32,107 characters)
    • bytes: 0x81 × 16,053 followed by 0x00 (about 16 KB of CBOR: an array of length 1, nested 16,053 times, around the integer 0)

The other six metadata entries are ordinary small objects and insert fine.

Root cause

  1. The ledger accepts arbitrarily deep metadata nesting, as long as the transaction fits the size limit. About 16 KB of 0x81 bytes gives about 16k levels.
  2. db-sync renders the metadata as JSON text and inserts it with json::jsonb in insertBulkTxMetadataPiped (Grouped.hs:345).
  3. PostgreSQL's JSON parser is recursive. 16k levels exceed the default 2 MB max_stack_depth, which raises SQLSTATE 54001.
  4. The error isn't handled. The block transaction rolls back, db-sync exits, and on restart it reprocesses the same block. This is a permanent crash loop.

Workarounds

  • Whitelist metadata labels so that label 1 is skipped: 
    "insert_options": { "tx_metadata": { "enable": true, "keys": [721, 674] } }
    You can also disable tx_metadata entirely.
  • Raise max_stack_depth in PostgreSQL. It must stay below the OS/container stack limit (ulimit -s), so this may also mean raising the container's stack ulimit. This only moves the limit: a deeper payload would break it again.

Suggested fix

db-sync should not crash on valid on-chain data. Possible approaches:

  • Check the nesting depth (or the JSON size) of the decoded metadata before the insert. When it is above a safe threshold, store json = NULL and keep the raw bytes. The json column is already nullable for metadata that can't be represented as JSON.
  • Or catch SQLSTATE 54001 on the tx_metadata insert, fall back to inserting the offending row with json = NULL, and log a warning.

Either way, the raw CBOR stays available in bytes and syncing continues.

Dominant language
Haskell
Stars
319
Forks
169
Avg merge
10h 54m
Merged PRs (30d)
3

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from IntersectMBO/cardano-db-sync

All issues in IntersectMBO/cardano-db-sync

Similar issues

More Haskell issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.