Unhandled inspector errors crash ShapeLogCollector; ReplicationClient then logs a full error every 50ms for up to 10 minutes
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- elixir, postgres
- Domain
- backend, databases, distributed-systems, observability-sre
Research direction
Start with the cited cases in packages/sync-service/lib/electric/shapes/partitions.ex and shape_log_collector.ex, then trace the retry paths in replication_client.ex and error handling in EtsInspector/DirectInspector. Review the existing add_shape and Inspector contracts before changing behavior. Done means inspector and restore failures no longer crash the replication pipeline, and retries avoid flooding logs or exposing full event payloads.
Written by the indexing model from the issue text.
Description
Summary
In Electric Cloud production (observed across sync-service 1.6.0–1.6.10) we see a steady stream of errors of the form:
Error processing replication event (NNNms retry budget left): ...
Error dispatching replication event (NNNms retry budget left): ** (exit) exited in: Electric.Replication.ShapeLogCollector.handle_event_async(...)
** (EXIT) no process: the process is not alive ...
Investigation shows these are one mechanism with several underlying gaps:
- The hot replication path (
ShapeLogCollector→Partitions→Inspector) does synchronous Postgres introspection but only handles one error value,{:error, :connection_not_available}. Every other legal return of the inspector contract crashes theShapeLogCollector. - The SLC crash takes down the whole
Electric.Shapes.Supervisor(:one_for_all), and while it restarts (which includes shape restore, potentially seconds to minutes),ReplicationClientretries the pending event every 50 ms, logging a full error — including the entire inspected event payload — on every attempt. A single incident can emit up to ~12,000 error log lines (10-minute budget / 50 ms), which flood logs and any error tracker attached to the logger.
Root causes (line refs at current main, 9b360dd9d)
1. Partitions.handle_relation is missing clauses for :table_not_found and generic {:error, reason}
packages/sync-service/lib/electric/shapes/partitions.ex — the case Inspector.load_relation_info(...) in handle_relation/2 (~L126) only matches {:ok, %{parent: ...}}, {:ok, _}, and {:error, :connection_not_available}.
But the inspector legitimately returns:
- bare
:table_not_found— e.g. when a table is dropped/renamed (or dropped+recreated with a new OID) between the WAL record being written and Electric introspecting it, such as while replaying backlog after a reconnect. Observed:{:case_clause, :table_not_found}crashes. Noteadd_shape/3in the same module does handle:table_not_found(~L67), so this looks like an omission. - in-band DB errors: the relation query stringifies errors (
{:error, Exception.message(err)}indirect_inspector.ex~L97), and the column query returns raw{:error, %Postgrex.Error{}}. Observed crashes include{:case_clause, {:error, "ERROR 53200 (out_of_memory) ..."}},{:case_clause, {:error, %Postgrex.Error{postgres: %{code: :query_canceled}}}}(source DBstatement_timeout), and{:case_clause, {:error, "ssl recv: closed"}}. I.e. any hiccup on the customer's database crashes Electric's replication pipeline.
2. pk_cols_of_relation leaks bare :table_not_found into Utils.map_while_ok
shape_log_collector.ex pk_cols_of_relation/2 is a with over Inspector.load_relation_oid/load_column_info; a bare :table_not_found falls through and reaches map_while_ok's mapper case (utils.ex:84), whose contract is {:ok, _} | {:error, _} → CaseClauseError. Same trigger scenario as above, different crash site.
3. Shape-restore path crashes on :connection_not_available
shape_log_collector.ex handle_continue(:restore_shapes) (~L282) does a bare match:
{:ok, partitions} = Partitions.add_shape(partitions, shape_handle, shape)
add_shape returns {:error, :connection_not_available} when the pool is unavailable, producing a badmatch crash during restart — i.e. exactly when the DB is likely still unhappy after one of the crashes above, turning one crash into a restart loop. The other add_shape call site (in the shape-registration update path) handles this value gracefully.
4. Connection-class in-band errors bypass the :connection_not_available classification
EtsInspector's wrap_in_db_errors only converts raised DBConnection.ConnectionError/checkout exits into {:error, :connection_not_available}. Connection errors returned in-band as tuples (e.g. "ssl recv: closed") flow through unclassified and hit either handle_relation's case_clause or add_shape's catch-all raise RuntimeError, "Unable to introspect table ..." (partitions.ex ~L75).
Amplification: an error logged every 50 ms, with the full event payload
replication_client.ex:
@event_retry_delay 50
@max_event_retry_time 10 * 60_000
Both retry paths (apply_event's catch for dispatch failures, and the :DOWN handler for handler crashes) log at error level on every attempt before re-scheduling 50 ms later. Until #4105 (1.6.0) these retries were completely silent (catch _, _ with no logging), so the pendulum swung from zero visibility to up to 20 error lines/second for up to 10 minutes per event.
Two extra problems with the log line itself:
- The dispatch-path message embeds
Exception.format(...)of the exit, which includes the fullinspected%TransactionFragment{}— i.e. complete customer row data ends up in logs (and in any log-capturing error tracker) on every retry. - Because the interpolated payload differs per event, error trackers can't group these — each incident shows up as a new issue.
Suggested shape of a fix for the logging: log the first failure at error (or warning), subsequent retries at debug or periodically (e.g. every 5–10 s), always log budget exhaustion at error; and log the event identity (xid/LSN/relation) rather than the full inspected payload.
Suggested fixes for the crashes
Partitions.handle_relation: handle:table_not_foundlikeadd_shapedoes, and decide an explicit policy for other{:error, _}(retryable vs. skip) instead of crashing.pk_cols_of_relation: normalize bare:table_not_foundbefore it reachesmap_while_ok.- Restore path: handle
{:error, :connection_not_available}fromadd_shape(retry/defer) instead of the bare{:ok, _} =match. EtsInspector/DirectInspector: classify connection-class in-band errors (TLS closed, etc.) as:connection_not_availableso the existing recovery path applies.
Context
- Retry loop introduced in #3097 (1.1.10, silent); per-attempt logging added in the async redesign in #4105 (1.6.0).
- All of the above verified present at current main (
9b360dd9d); the only post-1.6.7 change to the involved files is telemetry (#4504).
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 399
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valuePossibly taken @Tyagiquamar claimed this 19 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Possibly taken @mohanedmoh claimed this 37 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4826 · 1 comment ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakePossibly taken @adityavkk claimed this 133 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
electric-sql/electric#4687 ·
-
Listen IP addressOpenfeature request triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
electric-sql/electric#4150 · 1 comment ·
All issues in electric-sql/electric
Similar issues
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameOpenaccessibility
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
numerique-gouv/lasuite-landingpage#315 ·
Maintainers usually reply within 1 day
-
needs:triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
HyeokjaeChoi/vaultme#585 ·
Maintainers usually reply within 1 day
-
enhancement priority/P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
beettlle/pi-smart-router#200 ·