StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 42/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- elixir
- Domain
- backend, distributed-systems, performance
Research direction
Start by locating Electric.StatusMonitor, its wait_until/3 callers, and the existing wait_until_async/2 implementation. Read the current waiter and ETS readiness paths, then add Electric.PollWait with tests for readiness, timeout, backoff, and jitter. Done means the adaptive branch bounds GenServer replies during a delayed readiness flip while preserving the fast path.
Written by the indexing model from the issue text.
Description
Background
Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.
This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.
When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.
Goal
Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.
Design
1. A small reusable polling primitive
defmodule Electric.PollWait do
@moduledoc """
Per-process bounded polling of a cheap (ETS-backed) condition.
Sleeps between checks with exponential backoff (doubling, capped) and
bounded jitter so concurrent waiters land on distinct ETS reads
instead of stampeding the same millisecond windows.
"""
@default_initial_interval 25
@default_max_interval 500
@default_backoff 2.0
@default_jitter 0.25
@type check :: (-> {:ready, term()} | :ready | :not_ready)
@spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
def until(check_fun, timeout, opts \\ []) do
initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
max = Keyword.get(opts, :max_interval, @default_max_interval)
factor = Keyword.get(opts, :backoff, @default_backoff)
jitter = Keyword.get(opts, :jitter, @default_jitter)
do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
end
defp deadline(:infinity), do: :infinity
defp deadline(t) when is_integer(t) and t >= 0,
do: System.monotonic_time(:millisecond) + t
defp do_until(check_fun, deadline, interval, max, factor, jitter) do
case check_fun.() do
:not_ready ->
case remaining(deadline) do
0 -> :timeout
rem ->
Process.sleep(min(jittered(interval, jitter), rem))
do_until(check_fun, deadline, min(round(interval * factor), max),
max, factor, jitter)
end
ready -> ready
end
end
defp jittered(interval, jitter) do
spread = max(1, round(interval * jitter))
interval + :rand.uniform(2 * spread + 1) - spread - 1
end
defp remaining(:infinity), do: :infinity
defp remaining(deadline),
do: max(0, deadline - System.monotonic_time(:millisecond))
end
Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.
2. Adaptive switch-over driven by a congestion flag
We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.
Caller side:
def wait_until(stack_id, level, opts) do
case status_check(stack_id, level, opts) do
{:ready, value} ->
value
:not_ready ->
if congested?(stack_id),
do: poll_wait(stack_id, level, opts),
else: call_wait(stack_id, level, opts)
end
end
congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).
Server side:
@congested_threshold 100 # tunable
# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))
# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))
defp maybe_set_congested(state, size) when size >= @congested_threshold do
:ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok
defp maybe_clear_congested(state, 0) do
:ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok
Costs
Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.
Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.
Reusability
Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):
- Polling:
PollWait.until/3reuses verbatim. - Trigger:
GenServer.cast(EtsInspector, {:ensure_filled, key})to start the DB fill; cast is idempotent per key under server-side coalescing. - Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.
So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.
Suggested execution
- Land
Electric.PollWaitwith unit tests covering: ready/timeout/backoff schedule/jitter bounds. - Add the
@congested_keyflag + threshold logic toStatusMonitor. - Switch
wait_until/3callers to the adaptive branch. Delete the now-unusedwait_until_async/2if no callers remain. - Smoke-test the cold-start path: simulate N concurrent
wait_untilcalls during a delayedmark_*_readyflip and assert the GenServer's reply burst is bounded by@congested_threshold.
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 399
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valuePossibly taken @Tyagiquamar claimed this 19 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Possibly taken @mohanedmoh claimed this 37 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
electric-sql/electric#4826 · 1 comment ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakePossibly taken @adityavkk claimed this 133 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
electric-sql/electric#4687 ·
-
Listen IP addressOpenfeature request triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
electric-sql/electric#4150 · 1 comment ·
All issues in electric-sql/electric
Similar issues
-
area:cli area:security bug priority:medium
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameOpenaccessibility
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
numerique-gouv/lasuite-landingpage#315 ·
Maintainers usually reply within 1 day
-
needs:triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
HyeokjaeChoi/vaultme#585 ·
Maintainers usually reply within 1 day
-
enhancement priority/P2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
beettlle/pi-smart-router#200 ·