durable-streams-rust: log-structured per-stream data store to eliminate O(N) checkpoint fdatasync
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- rust
- Domain
- backend, data, performance
Research direction
Start with packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md and trace the WAL checkpoint path it describes. Define the segment layout, recovery/indexing behavior, and compaction path before implementation. Done means durability uses a bounded number of segment fsyncs rather than one fdatasync per touched stream, while reads and recovery remain functional.
Written by the indexing model from the issue text.
Description
Context
Investigation of the write cardinality cliff (throughput collapsing from ~279k ops/s at 100 streams to ~11k at 100k streams) traced the root cause to the WAL checkpoint: it makes each touched stream's data file durable via one fdatasync per stream — O(N_touched) device barriers (measured ~29s at 200k streams). That storm saturates the disk and stalls the commit-path fsyncs, which is the cliff.
Multi-lane hardware does NOT fix it: on c4d-standard-64-lssd with one physical NVMe device mounted per shard, throughput at 100k streams was flat across 1/2/4/6 lanes (~10-11k), because all shards' checkpoints fire simultaneously (shared ticker) and saturate every device at once.
Proposal
The O(N) cost exists because there is one file per stream. Move stream data into a bounded set of shared append-only segments (log-structured, like the WAL itself). Durability then costs O(segments) fsyncs, independent of stream count. Recovery replays/indexes the segments; a background compaction reclaims space via sequential rewrites + a handful of fsyncs.
- Pro: structurally eliminates the per-stream checkpoint storm; the cliff goes away.
- Con: per-stream reads become indexed segment lookups; requires a compaction path.
Alternatives tracked separately
- Incremental/staggered checkpoint +
--wal-checkpoint-syncfs(cheaper, amortize/coalesce the storm) — being prototyped now. - Reuse the existing cold tier as the durable bulk store (#COLD_TIER_ISSUE).
Findings: packages/durable-streams-rust/CARDINALITY_CLIFF_FIX.md.
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 399
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valuePossibly taken @Tyagiquamar claimed this 19 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Possibly taken @mohanedmoh claimed this 37 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4826 · 1 comment ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakePossibly taken @adityavkk claimed this 133 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
electric-sql/electric#4687 ·
-
Listen IP addressOpenfeature request triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
electric-sql/electric#4150 · 1 comment ·
All issues in electric-sql/electric
Similar issues
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameOpenaccessibility
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
numerique-gouv/lasuite-landingpage#315 ·
Maintainers usually reply within 1 day
-
needs:triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
HyeokjaeChoi/vaultme#585 ·
Maintainers usually reply within 1 day
-
enhancement priority/P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
beettlle/pi-smart-router#200 ·