durable-streams-rust: log-structured per-stream data store to eliminate O(N) checkpoint fdatasync

Open
#4,695 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
rust

Research direction

Start with packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md and trace the WAL checkpoint path it describes. Define the segment layout, recovery/indexing behavior, and compaction path before implementation. Done means durability uses a bounded number of segment fsyncs rather than one fdatasync per touched stream, while reads and recovery remain functional.

Written by the indexing model from the issue text.

Description

Context

Investigation of the write cardinality cliff (throughput collapsing from ~279k ops/s at 100 streams to ~11k at 100k streams) traced the root cause to the WAL checkpoint: it makes each touched stream's data file durable via one fdatasync per stream — O(N_touched) device barriers (measured ~29s at 200k streams). That storm saturates the disk and stalls the commit-path fsyncs, which is the cliff.

Multi-lane hardware does NOT fix it: on c4d-standard-64-lssd with one physical NVMe device mounted per shard, throughput at 100k streams was flat across 1/2/4/6 lanes (~10-11k), because all shards' checkpoints fire simultaneously (shared ticker) and saturate every device at once.

Proposal

The O(N) cost exists because there is one file per stream. Move stream data into a bounded set of shared append-only segments (log-structured, like the WAL itself). Durability then costs O(segments) fsyncs, independent of stream count. Recovery replays/indexes the segments; a background compaction reclaims space via sequential rewrites + a handful of fsyncs.

  • Pro: structurally eliminates the per-stream checkpoint storm; the cliff goes away.
  • Con: per-stream reads become indexed segment lookups; requires a compaction path.

Alternatives tracked separately

  • Incremental/staggered checkpoint + --wal-checkpoint-syncfs (cheaper, amortize/coalesce the storm) — being prototyped now.
  • Reuse the existing cold tier as the durable bulk store (#COLD_TIER_ISSUE).

Findings: packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md.

Dominant language
TypeScript
Stars
10.4k
Forks
399
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from electric-sql/electric

All issues in electric-sql/electric

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.