Failover/slot loss unhandled: reconnect loop never re-checks the slot, no timeline detection
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- postgresql, rust
- Domain
- databases, distributed-systems, observability
Research direction
Start with apps/engine/src/replication.rs:42-56 and trace setup_postgres and the reconnect loop. Compare the proposed behavior with timeline.ex, connection/manager.ex:452-500, and db_connection_error.ex. Done means reconnects detect system-id, timeline, or slot loss, purge shapes and force refetches, use backoff with jitter, and export a retained-WAL gauge.
Written by the indexing model from the issue text.
Description
Found during the production-readiness investigation.
Symptom / failure scenario: the ingestor's reconnect loop (apps/engine/src/replication.rs:42-56) retries pg::connect with a flat 500ms sleep and resumes peeking — ensure_slot runs only in setup_postgres at boot. After a failover to a promoted replica (or a dropped/invalidated slot), the slot doesn't exist: the loop errors forever. If the process is instead rebooted, ensure_slot silently creates a fresh slot at the new head LSN, losing every change in the gap — shapes are silently missing data with no signal.
There is also no timeline/system-id detection (upstream persists {pg_system_identifier, timeline_id} and purges all shapes on mismatch), and no exponential backoff.
Fix direction: persist system-id/timeline and compare at (re)connect → on mismatch or slot-gone, purge all shapes and force client refetch rather than continuing; exponential backoff with jitter; export a retained-WAL gauge. Reference: sync-service timeline.ex, connection/manager.ex:452-500, db_connection_error.ex.
🤖 Generated with Claude Code
- Dominant language
- Rust
- Stars
- 31
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Getting set up
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric-circuits
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
All issues in electric-sql/electric-circuits
Similar issues
-
area:casework bug criticality:p3 triage:needs-implementation
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
registrystack/registry-stack#1623 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
leptos-rs/leptos#4885 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
longbridge/gpui-kit#3276 ·
Maintainers usually reply within 1 day
-
A-Migration Guides D-Straightforward S-Ready-For-Implementation X-Uncontroversial
Difficulty 1/5 Under an hour Newbie friendliness 88/100
bevyengine/bevy-website#2607 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day