Postgres connection management: pool, streamed backfills, bounded concurrency, backoff, single-writer lock
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- postgresql, rust
- Domain
- backend, database, distributed-systems
Research direction
Start with the backfill connection code in apps/engine/src/engine.rs:1410-1413 and result handling in apps/engine/src/pg.rs:269-276, then read the ingestor reconnect loop in replication.rs:42-56. Done means the listed pool, streaming, concurrency, timeout, retry, error-classification, and single-writer behaviors are implemented without the explicitly out-of-scope admission control and TLS work.
Written by the indexing model from the issue text.
Description
Today every backfill opens a fresh Postgres connection (apps/engine/src/engine.rs:1410-1413) and collects the entire matching result set into a Vec<Row> (apps/engine/src/pg.rs:269-276); N concurrent shape creations = N connections + N full result sets in memory. The ingestor reconnect loop retries on a flat 500ms sleep (replication.rs:42-56).
Scope (aligned in discussion):
- Connection pool (e.g. deadpool-postgres) replacing connection-per-backfill; the ingestor keeps its dedicated session (slot reads are session-stateful).
- Streamed backfills: portal/
FETCH-based cursor reads, appended to the shape stream in chunks — no full-result-set materialization. Bounds memory per backfill regardless of shape breadth. - Bounded backfill concurrency: a semaphore on simultaneous snapshots (excess queue; no HTTP-layer admission control for now — explicitly out of scope, revisit under load testing).
- Slow-query protection: kill backfills that produce no first row within a timeout (upstream uses 30s) — protects the pool from index-less predicates.
- Exponential backoff (1s→10s, jittered) for all PG reconnects, replacing flat 500ms.
- Error taxonomy: classify PG errors retryable vs fatal-config (wrong wal_level, bad credentials, insufficient privileges) — fatal errors surface loudly instead of retry-looping. Upstream's
db_connection_error.exis a ready-made list. - Single-writer guard:
pg_advisory_lock(hashtext(slot_name))held by the ingestor — two engines peeking+advancing the same slot today would silently corrupt.
Out of scope: admission control / 503 shedding (deferred); TLS (tracked separately in its own issue).
🤖 Generated with Claude Code
- Dominant language
- Rust
- Stars
- 31
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric-circuits
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
DS disk growth: table-stream prefix trimming (log-native in ds-rust), size accounting; no compactionOpen
Difficulty 5/5 Over a week Newbie friendliness 38/100
All issues in electric-sql/electric-circuits
Similar issues
-
checkpoint/verify/rebuild_indexes on a thread holding a WriteTransaction deadlock instead of INT-001Openbug known-issue
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
project-minigraf/minigraf#734 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
bug CLI documentation
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
openai/codex#56630 · 3 comments · 3 reactions ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
DayuanJiang/PecoFence#139 ·
Maintainers usually reply within 1 day