StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 42/100
- Tipo di issue
- Refactoring
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- elixir
- Ambito
- backend, distributed-systems, performance
Direzione di ricerca
Inizia individuando Electric.StatusMonitor, i suoi chiamanti di wait_until/3 e l’implementazione esistente di wait_until_async/2. Leggi il waiter attuale e i percorsi di readiness di ETS, quindi aggiungi Electric.PollWait con test per readiness, timeout, backoff e jitter. Il lavoro è completo quando il ramo adattivo limita le risposte di GenServer durante un cambio di readiness ritardato, preservando al contempo il fast path.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Background
Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.
This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.
When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.
Goal
Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.
Design
1. A small reusable polling primitive
defmodule Electric.PollWait do
@moduledoc """
Per-process bounded polling of a cheap (ETS-backed) condition.
Sleeps between checks with exponential backoff (doubling, capped) and
bounded jitter so concurrent waiters land on distinct ETS reads
instead of stampeding the same millisecond windows.
"""
@default_initial_interval 25
@default_max_interval 500
@default_backoff 2.0
@default_jitter 0.25
@type check :: (-> {:ready, term()} | :ready | :not_ready)
@spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
def until(check_fun, timeout, opts \\ []) do
initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
max = Keyword.get(opts, :max_interval, @default_max_interval)
factor = Keyword.get(opts, :backoff, @default_backoff)
jitter = Keyword.get(opts, :jitter, @default_jitter)
do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
end
defp deadline(:infinity), do: :infinity
defp deadline(t) when is_integer(t) and t >= 0,
do: System.monotonic_time(:millisecond) + t
defp do_until(check_fun, deadline, interval, max, factor, jitter) do
case check_fun.() do
:not_ready ->
case remaining(deadline) do
0 -> :timeout
rem ->
Process.sleep(min(jittered(interval, jitter), rem))
do_until(check_fun, deadline, min(round(interval * factor), max),
max, factor, jitter)
end
ready -> ready
end
end
defp jittered(interval, jitter) do
spread = max(1, round(interval * jitter))
interval + :rand.uniform(2 * spread + 1) - spread - 1
end
defp remaining(:infinity), do: :infinity
defp remaining(deadline),
do: max(0, deadline - System.monotonic_time(:millisecond))
end
Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.
2. Adaptive switch-over driven by a congestion flag
We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.
Caller side:
def wait_until(stack_id, level, opts) do
case status_check(stack_id, level, opts) do
{:ready, value} ->
value
:not_ready ->
if congested?(stack_id),
do: poll_wait(stack_id, level, opts),
else: call_wait(stack_id, level, opts)
end
end
congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).
Server side:
@congested_threshold 100 # tunable
# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))
# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))
defp maybe_set_congested(state, size) when size >= @congested_threshold do
:ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok
defp maybe_clear_congested(state, 0) do
:ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok
Costs
Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.
Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.
Reusability
Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):
- Polling:
PollWait.until/3reuses verbatim. - Trigger:
GenServer.cast(EtsInspector, {:ensure_filled, key})to start the DB fill; cast is idempotent per key under server-side coalescing. - Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.
So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.
Suggested execution
- Land
Electric.PollWaitwith unit tests covering: ready/timeout/backoff schedule/jitter bounds. - Add the
@congested_keyflag + threshold logic toStatusMonitor. - Switch
wait_until/3callers to the adaptive branch. Delete the now-unusedwait_until_async/2if no callers remain. - Smoke-test the cold-start path: simulate N concurrent
wait_untilcalls during a delayedmark_*_readyflip and assert the GenServer's reply burst is bounded by@congested_threshold.
- Lingua principale
- TypeScript
- Stelle
- 10.4k
- Fork
- 399
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valueForse già presa @Tyagiquamar l’ha presa 19 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Forse già presa @mohanedmoh l’ha presa 37 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4826 · 1 commento ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakeForse già presa @adityavkk l’ha presa 133 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
electric-sql/electric#4687 ·
-
Listen IP addressApertafeature request triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
electric-sql/electric#4150 · 1 commento ·
Tutte le issue di electric-sql/electric
Issue simili
-
clawsweeper:fix-shape-clear clawsweeper:queueable-fix clawsweeper:source-repro impact:ux-friction issue-rating: 🦞 diamond lobster no-stale P2
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
openclaw/openclaw#199885 · 2 commenti · 1 reazione ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
-
feature needs-triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
BoardGamesEmpire/board-games-empire-backend#1306 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
sequant-io/sequant#1689 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100