Ops hardening: graceful shutdown, readiness probe, production metrics
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- postgresql, rust, typescript
- Domain
- backend, databases, infrastructure, observability
Research direction
Start with apps/engine/src/main.rs:67 and http.rs:16, then compare the existing TS ds-server/api-server SIGTERM handling and the /metrics/prometheus exporter. Break the work into shutdown, readiness, and production metrics, confirming the required PostgreSQL, replication, DS, request, backfill, shape, and ingest signals. Done means graceful SIGTERM behavior, distinct /health and gated /ready endpoints, and the listed metrics and OTel sampling controls on the utility port.
Written by the indexing model from the issue text.
Description
Graceful shutdown (engine): axum::serve has no with_graceful_shutdown and no SIGTERM handler (apps/engine/src/main.rs:67) — the process is hard-killed, dropping in-flight appends and long-polls (recovered only via at-least-once re-peek). Add SIGTERM handling: stop accepting, complete in-flight appends, close long-polls cleanly, stop the ingestor after its current batch (post-advance). The TS ds-server/api-server already handle SIGTERM.
Readiness vs liveness: /health returns a static "ok" before Postgres or DS are reachable (http.rs:16). Add /ready gated on {PG connected, slot exists and streaming, DS reachable}; keep /health as pure liveness. Optionally upstream's hold-briefly-then-503 pattern for requests arriving during startup (5s, then retry-after).
Metrics (gaps vs the set upstream exports; ours cover engine internals well but not ops):
slot_retained_wal_size+confirmed_flush_lsn_lag(bytes) — the two that page you before the primary's disk fills- replication receive-lag distribution (commit-to-ingest ms)
- request counts by status (
/v1/shape+ control plane), tagged known-error vs unexpected - backfill duration/bytes/rows histograms; per-shape append-failure counter
- active vs total shapes (with the #9 tiers: active/dormant), DS disk usage per class (
table/*vsshape/*) - error-rate counters on ingest decode/append paths
These land on the utility port (see auth issue). Wire the existing /metrics/prometheus exporter; add OTel trace-sampling knobs before enabling tracing at production volume (upstream defaults to 1% span sampling).
🤖 Generated with Claude Code
- Dominant language
- Rust
- Stars
- 31
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric-circuits
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
DS disk growth: table-stream prefix trimming (log-native in ds-rust), size accounting; no compactionOpen
Difficulty 5/5 Over a week Newbie friendliness 38/100
All issues in electric-sql/electric-circuits
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
pedrovgs/UseCaseCoverage#23 ·
-
priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
storytold/photocraft#8687 · 1 comment ·
Maintainers usually reply within 2 days
-
area:kernel bug kernel-paused needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
bherbruck/solvecraft#5055 · 2 comments ·
-
bug cli p3
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 3 days
-
documentation good first issue help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day