Unbounded memory on large transactions: peek cap escalates until the whole txn fits in RAM
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start in apps/engine/src/replication.rs:76-166 to trace decode_batch's PEAK_CAP growth and tx_envs buffering, then inspect the tailer's staging in engine.rs:1252-1276. Compare the proposed guard with upstream's message_converter.ex:96-102. Done should prevent large transactions from causing an OOM and define the affected-shape recovery behavior, or clearly scope the spill/streaming work separately.
Written by the indexing model from the issue text.
Description
Found during the production-readiness investigation.
Symptom / failure scenario: decode_batch buffers a whole transaction in tx_envs to stamp commit lsn/xid/seq, and when a transaction doesn't fit in PEEK_CAP (5000 rows) the cap escalates ×4 repeatedly until it does (apps/engine/src/replication.rs:76-166). Under REPLICA IDENTITY FULL each change carries old+new tuples, and the raw peek strings + decoded envelopes + the tailer's per-shape staging (engine.rs:1252-1276) coexist. A 1M-row UPDATE is a multi-hundred-MB-to-GB transient spike — the engine OOMs instead of degrading.
Fix direction (staged):
- Guard now: a max-txn-size limit that fails loudly (invalidate affected shapes → client refetch) instead of OOM. Upstream uses 250 MiB (
message_converter.ex:96-102). - Spill: past a threshold, spill
tx_envsto a temp file, stream through stamping on COMMIT. - Real fix: streaming logical replication with pgoutput proto v2+ (PG 14+), which streams in-progress large transactions — eliminates whole-txn buffering at the source (upstream hasn't done this either; their spill is still a TODO).
🤖 Generated with Claude Code
- Dominant language
- Rust
- Stars
- 31
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric-circuits
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
All issues in electric-sql/electric-circuits
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
bytecodealliance/wasmtime#14433 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day