StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling

オープン
#4,371 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
42/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
静か
技術スタック
elixir

調査の方向性

まず Electric.StatusMonitor、wait_until/3 の呼び出し元、および既存の wait_until_async/2 の実装を見つけます。現在の waiter と ETS の準備完了経路を読み、その後、準備完了、タイムアウト、バックオフ、ジッターのテストとともに Electric.PollWait を追加します。遅延した準備完了状態の切り替え中に adaptive ブランチが GenServer の応答を制限し、fast path を維持できれば完了です。

索引モデルが issue の本文から書いたものです。

説明

Background

Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.

This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.

When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.

Goal

Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.

Design

1. A small reusable polling primitive
defmodule Electric.PollWait do
  @moduledoc """
  Per-process bounded polling of a cheap (ETS-backed) condition.

  Sleeps between checks with exponential backoff (doubling, capped) and
  bounded jitter so concurrent waiters land on distinct ETS reads
  instead of stampeding the same millisecond windows.
  """

  @default_initial_interval 25
  @default_max_interval 500
  @default_backoff 2.0
  @default_jitter 0.25

  @type check :: (-> {:ready, term()} | :ready | :not_ready)

  @spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
  def until(check_fun, timeout, opts \\ []) do
    initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
    max     = Keyword.get(opts, :max_interval, @default_max_interval)
    factor  = Keyword.get(opts, :backoff, @default_backoff)
    jitter  = Keyword.get(opts, :jitter, @default_jitter)

    do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
  end

  defp deadline(:infinity), do: :infinity
  defp deadline(t) when is_integer(t) and t >= 0,
    do: System.monotonic_time(:millisecond) + t

  defp do_until(check_fun, deadline, interval, max, factor, jitter) do
    case check_fun.() do
      :not_ready ->
        case remaining(deadline) do
          0 -> :timeout
          rem ->
            Process.sleep(min(jittered(interval, jitter), rem))
            do_until(check_fun, deadline, min(round(interval * factor), max),
                     max, factor, jitter)
        end

      ready -> ready
    end
  end

  defp jittered(interval, jitter) do
    spread = max(1, round(interval * jitter))
    interval + :rand.uniform(2 * spread + 1) - spread - 1
  end

  defp remaining(:infinity), do: :infinity
  defp remaining(deadline),
    do: max(0, deadline - System.monotonic_time(:millisecond))
end

Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.

2. Adaptive switch-over driven by a congestion flag

We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.

Caller side:

def wait_until(stack_id, level, opts) do
  case status_check(stack_id, level, opts) do
    {:ready, value} ->
      value

    :not_ready ->
      if congested?(stack_id),
        do: poll_wait(stack_id, level, opts),
        else: call_wait(stack_id, level, opts)
  end
end

congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).

Server side:

@congested_threshold 100   # tunable

# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))

# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))

defp maybe_set_congested(state, size) when size >= @congested_threshold do
  :ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok

defp maybe_clear_congested(state, 0) do
  :ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok

Costs

Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.

Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.

Reusability

Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):

  • Polling: PollWait.until/3 reuses verbatim.
  • Trigger: GenServer.cast(EtsInspector, {:ensure_filled, key}) to start the DB fill; cast is idempotent per key under server-side coalescing.
  • Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.

So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.

Suggested execution

  1. Land Electric.PollWait with unit tests covering: ready/timeout/backoff schedule/jitter bounds.
  2. Add the @congested_key flag + threshold logic to StatusMonitor.
  3. Switch wait_until/3 callers to the adaptive branch. Delete the now-unused wait_until_async/2 if no callers remain.
  4. Smoke-test the cold-start path: simulate N concurrent wait_until calls during a delayed mark_*_ready flip and assert the GenServer's reply burst is bounded by @congested_threshold.
主要言語
TypeScript
スター
10.4k
フォーク
399
PR マージ指標
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

electric-sql/electric のほかの issue

electric-sql/electric の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。