StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling

Open
#4,371 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
42/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Quiet
Tech stack
elixir

Research direction

Start by locating Electric.StatusMonitor, its wait_until/3 callers, and the existing wait_until_async/2 implementation. Read the current waiter and ETS readiness paths, then add Electric.PollWait with tests for readiness, timeout, backoff, and jitter. Done means the adaptive branch bounds GenServer replies during a delayed readiness flip while preserving the fast path.

Written by the indexing model from the issue text.

Description

Background

Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.

This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.

When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.

Goal

Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.

Design

1. A small reusable polling primitive
defmodule Electric.PollWait do
  @moduledoc """
  Per-process bounded polling of a cheap (ETS-backed) condition.

  Sleeps between checks with exponential backoff (doubling, capped) and
  bounded jitter so concurrent waiters land on distinct ETS reads
  instead of stampeding the same millisecond windows.
  """

  @default_initial_interval 25
  @default_max_interval 500
  @default_backoff 2.0
  @default_jitter 0.25

  @type check :: (-> {:ready, term()} | :ready | :not_ready)

  @spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
  def until(check_fun, timeout, opts \\ []) do
    initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
    max     = Keyword.get(opts, :max_interval, @default_max_interval)
    factor  = Keyword.get(opts, :backoff, @default_backoff)
    jitter  = Keyword.get(opts, :jitter, @default_jitter)

    do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
  end

  defp deadline(:infinity), do: :infinity
  defp deadline(t) when is_integer(t) and t >= 0,
    do: System.monotonic_time(:millisecond) + t

  defp do_until(check_fun, deadline, interval, max, factor, jitter) do
    case check_fun.() do
      :not_ready ->
        case remaining(deadline) do
          0 -> :timeout
          rem ->
            Process.sleep(min(jittered(interval, jitter), rem))
            do_until(check_fun, deadline, min(round(interval * factor), max),
                     max, factor, jitter)
        end

      ready -> ready
    end
  end

  defp jittered(interval, jitter) do
    spread = max(1, round(interval * jitter))
    interval + :rand.uniform(2 * spread + 1) - spread - 1
  end

  defp remaining(:infinity), do: :infinity
  defp remaining(deadline),
    do: max(0, deadline - System.monotonic_time(:millisecond))
end

Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.

2. Adaptive switch-over driven by a congestion flag

We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.

Caller side:

def wait_until(stack_id, level, opts) do
  case status_check(stack_id, level, opts) do
    {:ready, value} ->
      value

    :not_ready ->
      if congested?(stack_id),
        do: poll_wait(stack_id, level, opts),
        else: call_wait(stack_id, level, opts)
  end
end

congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).

Server side:

@congested_threshold 100   # tunable

# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))

# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))

defp maybe_set_congested(state, size) when size >= @congested_threshold do
  :ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok

defp maybe_clear_congested(state, 0) do
  :ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok

Costs

Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.

Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.

Reusability

Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):

  • Polling: PollWait.until/3 reuses verbatim.
  • Trigger: GenServer.cast(EtsInspector, {:ensure_filled, key}) to start the DB fill; cast is idempotent per key under server-side coalescing.
  • Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.

So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.

Suggested execution

  1. Land Electric.PollWait with unit tests covering: ready/timeout/backoff schedule/jitter bounds.
  2. Add the @congested_key flag + threshold logic to StatusMonitor.
  3. Switch wait_until/3 callers to the adaptive branch. Delete the now-unused wait_until_async/2 if no callers remain.
  4. Smoke-test the cold-start path: simulate N concurrent wait_until calls during a delayed mark_*_ready flip and assert the GenServer's reply burst is bounded by @congested_threshold.
Dominant language
TypeScript
Stars
10.4k
Forks
399
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from electric-sql/electric

All issues in electric-sql/electric

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.