durable-streams-rust: use the cold tier as the durable bulk store (make local per-stream files a non-durable read cache)

Open
#4,696 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Quiet
Tech stack
aws

Research direction

Start with CARDINALITY_CLIFF_FIX.md and the WAL checkpoint and cold-tier entry points exposed by the --tier-* options. Trace how tier uploads, WAL recycling, and recovery ordering interact, including --tier-segment-bytes and --tier-compact-bytes. Done means the open recovery and durability questions are resolved and the proposed cache-only local files no longer require an O(N) durability checkpoint.

Written by the indexing model from the issue text.

Description

Context

The write cardinality cliff is caused by the WAL checkpoint's O(N_touched) per-stream fdatasync storm (see investigation in CARDINALITY_CLIFF_FIX.md). The checkpoint exists only to make per-stream data durable so the WAL can be recycled.

Proposal

The server already has a cold tier (S3/MinIO segments, --tier-*). Make the local per-stream files a pure read cache, not a durability target:

  • Durability = WAL (recent) + cold-tier segments (bulk).
  • The WAL recycles once its records are persisted to the tier — which is already batched into bounded segments, i.e. O(segments) uploads/fsyncs, not O(streams).
  • Recovery = load tier state + replay the WAL tail. The per-stream local files are rebuilt lazily as a cache and never need a per-file fdatasync.

This deletes the O(N) checkpoint by reusing infrastructure that already exists, and fits the tiered architecture better than one-file-per-stream local durability.

Open questions

  • Is the cold tier currently authoritative enough for recovery (upload durability, ordering vs WAL recycle floor)?
  • Interaction with --tier-segment-bytes / --tier-compact-bytes cadence.

Alternatives tracked separately

  • Log-structured local store (#LOG_STRUCTURED_ISSUE).
  • Incremental/staggered checkpoint + --wal-checkpoint-syncfs (being prototyped now).
Dominant language
TypeScript
Stars
10.4k
Forks
399
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from electric-sql/electric

All issues in electric-sql/electric

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.