StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling

Aperta
#4,371 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
42/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
elixir

Direzione di ricerca

Inizia individuando Electric.StatusMonitor, i suoi chiamanti di wait_until/3 e l’implementazione esistente di wait_until_async/2. Leggi il waiter attuale e i percorsi di readiness di ETS, quindi aggiungi Electric.PollWait con test per readiness, timeout, backoff e jitter. Il lavoro è completo quando il ramo adattivo limita le risposte di GenServer durante un cambio di readiness ritardato, preservando al contempo il fast path.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Background

Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.

This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.

When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.

Goal

Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.

Design

1. A small reusable polling primitive
defmodule Electric.PollWait do
  @moduledoc """
  Per-process bounded polling of a cheap (ETS-backed) condition.

  Sleeps between checks with exponential backoff (doubling, capped) and
  bounded jitter so concurrent waiters land on distinct ETS reads
  instead of stampeding the same millisecond windows.
  """

  @default_initial_interval 25
  @default_max_interval 500
  @default_backoff 2.0
  @default_jitter 0.25

  @type check :: (-> {:ready, term()} | :ready | :not_ready)

  @spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
  def until(check_fun, timeout, opts \\ []) do
    initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
    max     = Keyword.get(opts, :max_interval, @default_max_interval)
    factor  = Keyword.get(opts, :backoff, @default_backoff)
    jitter  = Keyword.get(opts, :jitter, @default_jitter)

    do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
  end

  defp deadline(:infinity), do: :infinity
  defp deadline(t) when is_integer(t) and t >= 0,
    do: System.monotonic_time(:millisecond) + t

  defp do_until(check_fun, deadline, interval, max, factor, jitter) do
    case check_fun.() do
      :not_ready ->
        case remaining(deadline) do
          0 -> :timeout
          rem ->
            Process.sleep(min(jittered(interval, jitter), rem))
            do_until(check_fun, deadline, min(round(interval * factor), max),
                     max, factor, jitter)
        end

      ready -> ready
    end
  end

  defp jittered(interval, jitter) do
    spread = max(1, round(interval * jitter))
    interval + :rand.uniform(2 * spread + 1) - spread - 1
  end

  defp remaining(:infinity), do: :infinity
  defp remaining(deadline),
    do: max(0, deadline - System.monotonic_time(:millisecond))
end

Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.

2. Adaptive switch-over driven by a congestion flag

We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.

Caller side:

def wait_until(stack_id, level, opts) do
  case status_check(stack_id, level, opts) do
    {:ready, value} ->
      value

    :not_ready ->
      if congested?(stack_id),
        do: poll_wait(stack_id, level, opts),
        else: call_wait(stack_id, level, opts)
  end
end

congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).

Server side:

@congested_threshold 100   # tunable

# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))

# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))

defp maybe_set_congested(state, size) when size >= @congested_threshold do
  :ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok

defp maybe_clear_congested(state, 0) do
  :ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok

Costs

Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.

Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.

Reusability

Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):

  • Polling: PollWait.until/3 reuses verbatim.
  • Trigger: GenServer.cast(EtsInspector, {:ensure_filled, key}) to start the DB fill; cast is idempotent per key under server-side coalescing.
  • Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.

So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.

Suggested execution

  1. Land Electric.PollWait with unit tests covering: ready/timeout/backoff schedule/jitter bounds.
  2. Add the @congested_key flag + threshold logic to StatusMonitor.
  3. Switch wait_until/3 callers to the adaptive branch. Delete the now-unused wait_until_async/2 if no callers remain.
  4. Smoke-test the cold-start path: simulate N concurrent wait_until calls during a delayed mark_*_ready flip and assert the GenServer's reply burst is bounded by @congested_threshold.
Lingua principale
TypeScript
Stelle
10.4k
Fork
399
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di electric-sql/electric

Tutte le issue di electric-sql/electric

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.