StatusMonitor: replace mailbox-based wait_until with adaptive per-process polling
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 42/100
- issue の種類
- リファクタリング
- 明瞭さ
- おおむね明確
- 活発さ
- 静か
- 技術スタック
- elixir
調査の方向性
まず Electric.StatusMonitor、wait_until/3 の呼び出し元、および既存の wait_until_async/2 の実装を見つけます。現在の waiter と ETS の準備完了経路を読み、その後、準備完了、タイムアウト、バックオフ、ジッターのテストとともに Electric.PollWait を追加します。遅延した準備完了状態の切り替え中に adaptive ブランチが GenServer の応答を制限し、fast path を維持できれば完了です。
索引モデルが issue の本文から書いたものです。
説明
Background
Electric.StatusMonitor.wait_until/3 is called from Api.validate_params → hold_until_stack_ready on every shape HTTP request. When the stack is already ready, the call short-circuits on a pure ETS read of the per-stack status table — fine. When the stack is not yet ready (cold start, post-deploy, recovering from connection sleep, transient readiness flip), the call falls through to GenServer.call(StatusMonitor, {:wait_until, level, timeout}, :infinity) and parks in the StatusMonitor's waiters MapSet until readiness is signalled.
This is a follow-up to the thundering-herd mitigation work tracked under #4266 — same family as the cheap admission control (#4292 / #4291 / PR #4359) and the EtsInspector mailbox-overload issue (#4370). The pattern is the same: a single GenServer fronting a cheap, ETS-observable signal, with :infinity per-waiter mailbox parking that doesn't scale to the concurrent populations admission control now admits.
When a new instance has just started and the stack is not ready while thousands of shape requests arrive, the StatusMonitor accumulates thousands of waiters. The readiness flip then triggers maybe_reply_to_waiters/1 to iterate every entry and GenServer.reply/2 to each — a serial reply burst from a single process, on top of whatever other mailbox traffic is in flight.
Goal
Convert the not-ready path from "park in a single GenServer mailbox" to "each request waits on its own using cheap ETS polling", while keeping the fast path identical and preserving low-latency wakeup in the uncongested case.
Design
1. A small reusable polling primitive
defmodule Electric.PollWait do
@moduledoc """
Per-process bounded polling of a cheap (ETS-backed) condition.
Sleeps between checks with exponential backoff (doubling, capped) and
bounded jitter so concurrent waiters land on distinct ETS reads
instead of stampeding the same millisecond windows.
"""
@default_initial_interval 25
@default_max_interval 500
@default_backoff 2.0
@default_jitter 0.25
@type check :: (-> {:ready, term()} | :ready | :not_ready)
@spec until(check, timeout(), keyword()) :: {:ready, term()} | :ready | :timeout
def until(check_fun, timeout, opts \\ []) do
initial = Keyword.get(opts, :initial_interval, @default_initial_interval)
max = Keyword.get(opts, :max_interval, @default_max_interval)
factor = Keyword.get(opts, :backoff, @default_backoff)
jitter = Keyword.get(opts, :jitter, @default_jitter)
do_until(check_fun, deadline(timeout), initial, max, factor, jitter)
end
defp deadline(:infinity), do: :infinity
defp deadline(t) when is_integer(t) and t >= 0,
do: System.monotonic_time(:millisecond) + t
defp do_until(check_fun, deadline, interval, max, factor, jitter) do
case check_fun.() do
:not_ready ->
case remaining(deadline) do
0 -> :timeout
rem ->
Process.sleep(min(jittered(interval, jitter), rem))
do_until(check_fun, deadline, min(round(interval * factor), max),
max, factor, jitter)
end
ready -> ready
end
end
defp jittered(interval, jitter) do
spread = max(1, round(interval * jitter))
interval + :rand.uniform(2 * spread + 1) - spread - 1
end
defp remaining(:infinity), do: :infinity
defp remaining(deadline),
do: max(0, deadline - System.monotonic_time(:millisecond))
end
Default backoff schedule: 25 → 50 → 100 → 200 → 400 → 500 → 500 … ms, with ±25% jitter per step. A waiter for a stack that takes 30s to come up does ~80 ETS reads spread across distinct millisecond windows, not 600+ stampeded ones.
2. Adaptive switch-over driven by a congestion flag
We do NOT want to pay the polling latency cost in the common-case where the GenServer mailbox is fine. The switch-over is opt-in based on whether the StatusMonitor itself reports that its waiter set has crossed a threshold.
Caller side:
def wait_until(stack_id, level, opts) do
case status_check(stack_id, level, opts) do
{:ready, value} ->
value
:not_ready ->
if congested?(stack_id),
do: poll_wait(stack_id, level, opts),
else: call_wait(stack_id, level, opts)
end
end
congested?/1 is one ETS read on the same StatusMonitor:<stack_id> table that already backs the fast-path readiness check (already configured with read_concurrency: true).
Server side:
@congested_threshold 100 # tunable
# In handle_call({:wait_until, ...}), after MapSet.put:
new_waiters = MapSet.put(waiters, {from, level})
maybe_set_congested(state, MapSet.size(new_waiters))
# In maybe_reply_to_waiters/1, after the reduce:
maybe_clear_congested(state, MapSet.size(remaining_waiters))
defp maybe_set_congested(state, size) when size >= @congested_threshold do
:ets.insert(ets_table(state.stack_id), {@congested_key, true})
end
defp maybe_set_congested(_, _), do: :ok
defp maybe_clear_congested(state, 0) do
:ets.insert(ets_table(state.stack_id), {@congested_key, false})
end
defp maybe_clear_congested(_, _), do: :ok
Costs
Per blocked request in the polling path: ~6–8 ETS reads spread over a 30s wait (because of doubling backoff). The StatusMonitor ETS table is already read_concurrency: true so concurrent reads scale.
Wake-up latency for polled waiters: bounded by the current backoff step (max 500ms once steady state is reached, with jitter). The readiness flip happens once at startup or after a recovery, so the latency budget is trivially absorbed by the request's own HTTP/admission timeout. In exchange, the StatusMonitor no longer has to serially GenServer.reply to thousands of waiters.
Reusability
Same PollWait + congestion-flag pattern slots into the EtsInspector follow-up (#4370):
- Polling:
PollWait.until/3reuses verbatim. - Trigger:
GenServer.cast(EtsInspector, {:ensure_filled, key})to start the DB fill; cast is idempotent per key under server-side coalescing. - Congestion flag: same ETS-flag mechanism, sized for the inspector's mailbox.
So this issue lands the primitive + StatusMonitor cutover, and #4370 reuses it.
Suggested execution
- Land
Electric.PollWaitwith unit tests covering: ready/timeout/backoff schedule/jitter bounds. - Add the
@congested_keyflag + threshold logic toStatusMonitor. - Switch
wait_until/3callers to the adaptive branch. Delete the now-unusedwait_until_async/2if no callers remain. - Smoke-test the cold-start path: simulate N concurrent
wait_untilcalls during a delayedmark_*_readyflip and assert the GenServer's reply burst is bounded by@congested_threshold.
- 主要言語
- TypeScript
- スター
- 10.4k
- フォーク
- 399
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
electric-sql/electric のほかの issue
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryオープン
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error value対応中かも @Tyagiquamar が 19 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)対応中かも @mohanedmoh が 37 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
electric-sql/electric#4826 · コメント 1 件 ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wake対応中かも @adityavkk が 133 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
electric-sql/electric#4687 ·
-
Listen IP addressオープンfeature request triage
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
electric-sql/electric#4150 · コメント 1 件 ·
electric-sql/electric の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
tintinweb/pi-subagents#459 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
danielmiessler/LifeOS#2830 ·
メンテナーはふだん 6 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
freshframework/fresh#3920 ·
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
area: client enhancement priority: low
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
opensro-dev/opensro#3681 ·
メンテナーはふだん 1 日以内に返信