durable-streams-rust: use the cold tier as the durable bulk store (make local per-stream files a non-durable read cache)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 28/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- aws
- Domain
- backend, distributed-systems
Research direction
Start with CARDINALITY_CLIFF_FIX.md and the WAL checkpoint and cold-tier entry points exposed by the --tier-* options. Trace how tier uploads, WAL recycling, and recovery ordering interact, including --tier-segment-bytes and --tier-compact-bytes. Done means the open recovery and durability questions are resolved and the proposed cache-only local files no longer require an O(N) durability checkpoint.
Written by the indexing model from the issue text.
Description
Context
The write cardinality cliff is caused by the WAL checkpoint's O(N_touched) per-stream fdatasync storm (see investigation in CARDINALITY_CLIFF_FIX.md). The checkpoint exists only to make per-stream data durable so the WAL can be recycled.
Proposal
The server already has a cold tier (S3/MinIO segments, --tier-*). Make the local per-stream files a pure read cache, not a durability target:
- Durability = WAL (recent) + cold-tier segments (bulk).
- The WAL recycles once its records are persisted to the tier — which is already batched into bounded segments, i.e.
O(segments)uploads/fsyncs, notO(streams). - Recovery = load tier state + replay the WAL tail. The per-stream local files are rebuilt lazily as a cache and never need a per-file
fdatasync.
This deletes the O(N) checkpoint by reusing infrastructure that already exists, and fits the tiered architecture better than one-file-per-stream local durability.
Open questions
- Is the cold tier currently authoritative enough for recovery (upload durability, ordering vs WAL recycle floor)?
- Interaction with
--tier-segment-bytes/--tier-compact-bytescadence.
Alternatives tracked separately
- Log-structured local store (#LOG_STRUCTURED_ISSUE).
- Incremental/staggered checkpoint +
--wal-checkpoint-syncfs(being prototyped now).
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 399
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valuePossibly taken @Tyagiquamar claimed this 19 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Possibly taken @mohanedmoh claimed this 37 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4826 · 1 comment ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakePossibly taken @adityavkk claimed this 133 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
electric-sql/electric#4687 ·
-
Listen IP addressOpenfeature request triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
electric-sql/electric#4150 · 1 comment ·
All issues in electric-sql/electric
Similar issues
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameOpenaccessibility
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
numerique-gouv/lasuite-landingpage#315 ·
Maintainers usually reply within 1 day
-
needs:triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
HyeokjaeChoi/vaultme#585 ·
Maintainers usually reply within 1 day
-
enhancement priority/P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
beettlle/pi-smart-router#200 ·