ShapeCache: coalesce concurrent create_or_wait callers via GenServer-owned per-shape lock + polling
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
- Tipo di issue
- Funzionalità
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- elixir
- Ambito
- backend, distributed-systems
Direzione di ricerca
Inizia da Electric.ShapeCache.get_or_create_shape_handle/3, dal suo handler create_or_wait_shape_handle e da ShapeCache init/1; esamina prima l’API PollWait di #4371. Traccia il flusso esistente di safe_maybe_create_shape e le tabelle ETS prima di aggiungere il lock per shape e il percorso di polling. Il lavoro è completato quando i caller della stessa shape evitano l’accumulo nella mailbox, il recupero da crash cancella i lock e passano i test di concorrenza, backoff e riavvio.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Background
Electric.ShapeCache.get_or_create_shape_handle/3 is called from Api.load_shape_info on every shape request that misses the in-process fast path. The fast path checks ShapeStatus.fetch_handle_by_shape (WriteBuffer ETS, then SQLite read connection) followed by fetch_latest_offset (Storage). On miss, the caller falls through to GenServer.call(ShapeCache, {:create_or_wait_shape_handle, …}, 30_000).
When N concurrent requests for the same uncached shape arrive, all of them miss the fast path and all of them enqueue a GenServer.call. Handler 1 does the actual creation work (safe_maybe_create_shape → add_shape + start_shape); handlers 2..N short-circuit via fetch_handle_by_shape_critical finding the shape that handler 1 just created. So the work is deduplicated, but every caller still occupies a mailbox slot for the full duration of handler 1's creation plus its own dispatch latency. Under filesystem tail latency, shape creation reaches multiple seconds and the queue depth is real.
This is a sibling of #4370 (EtsInspector) and #4371 (StatusMonitor) under the thundering-herd umbrella #4266. The cheap admission control (#4292 / #4291 / PR #4359) bounds the concurrent population that can reach ShapeCache at all via the :initial bucket, but does not prevent admitted requests for the same shape from piling into the ShapeCache mailbox.
Goal
Prevent the ShapeCache mailbox from accumulating concurrent calls for the same fresh shape. Only one caller per unique shape-creation event should issue a GenServer.call; every other caller should observe the in-progress state via ETS and poll until the result is ready.
Design
The lock is GenServer-owned for writes, caller-readable for routing decisions. Callers never write to the lock table, eliminating any stale-claim concerns.
Caller flow
1. fast_path(shape) hit? → return
2. :ets.member(shape_create_lock(stack_id), shape_hash)?
true → PollWait.until(poll_predicate, @call_timeout, <ShapeCache opts>)
false → GenServer.call(name(stack_id), {:create_or_wait_shape_handle, …}, @call_timeout)
Server handler (replaces today's body)
case fetch_handle_by_shape_critical(stack_id, shape) do
{:ok, handle} -> reply {handle, fetch_latest_offset(...)}
:error ->
:ets.insert(shape_create_lock(stack_id), {shape_hash, self()})
result = safe_maybe_create_shape(shape, state)
:ets.delete(shape_create_lock(stack_id), shape_hash)
reply result
end
Poll predicate
with {:ok, handle} <- ShapeStatus.fetch_handle_by_shape(stack_id, shape),
true <- ShapeStatus.shape_has_been_activated?(stack_id, handle),
{:ok, offset} <- fetch_latest_offset(stack_id, handle) do
{:ready, {handle, offset}}
else
_ -> :not_ready
end
shape_has_been_activated?/2 checks the last_used_timestamp that update_last_read_time_to_now/2 sets at the very end of start_shape, so it serves as a "creation fully succeeded" signal. If creation fails, clean_shape removes the row and waiters never observe :ready, eventually timing out.
Properties
- Mailbox growth bounded by a single race window. Once a shape's lock is set, all new callers bypass the GenServer entirely. The mailbox grows only during the window between "first caller's call queued" and "handler dequeues and sets the lock" — i.e. mailbox dispatch latency in the common case, the previous shape's creation time in the contended case (and even then, capped at "one entry per fast-path-miss within the window").
- No
:waitreply needed. Any caller whose call reaches the handler with the lock unset is either the creator (does the work) or finds the shape already exists via the critical fetch (reply with the cached result). There's no third response code. - No stale-lock recovery needed. Callers don't write the lock, so they can't strand it. The GenServer always clears the lock in the same handler invocation that set it. On GenServer restart,
init/1wipes the lock table. - Specific error reasons are lost for polling waiters. They get
:timeoutafter the deadline. Same trade-off as #4370 / #4371.
Backoff considerations — PollWait must accept per-caller config
This is the part to be aware of when implementing.
The defaults in #4371 (initial 25ms → doubling → 500ms cap) are tuned for stack-readiness waiting, where the underlying event flips on a second-to-minute timescale and aggressive polling is wasted. ShapeCache is the opposite: a single shape creation completes in tens to low-hundreds of milliseconds in the common case. Polling at a 500ms cap would routinely add hundreds of milliseconds of latency for no benefit.
ShapeCache's caller should pass its own backoff:
PollWait.until(poll_predicate, @call_timeout,
initial_interval: 5,
max_interval: 100,
backoff: 2.0
)
Schedule: 5 → 10 → 20 → 40 → 80 → 100 → 100 … ms. Most shape creations are answered within the first 2–3 polls.
The PollWait primitive proposed in #4371 already accepts these as opts, so no primitive change is needed — only that the implementation honour the per-call overrides without baking the StatusMonitor defaults into a global config.
Crash safety
If the GenServer crashes mid-creation, its state is lost but the named, public lock table persists. On restart, init/1 runs :ets.delete_all_objects(shape_create_lock(stack_id)) to wipe stale entries. Any in-flight waiters that were polling time out after their existing deadline; new requests start fresh.
Suggested execution
- Depends on
Electric.PollWaitlanding as part of #4371. - Add the per-shape lock ETS table (
shape_create_lock:<stack_id>), created in ShapeCache'sinit/1with:public, :set, :named_table, read_concurrency: true, write_concurrency: :auto. Cleared ininit/1to handle crash recovery. - Modify
get_or_create_shape_handle/3to add the:ets.membercheck beforeGenServer.call, with the polling fallback. - Modify the
handle_call({:create_or_wait_shape_handle, …}, …)body to set/clear the lock aroundsafe_maybe_create_shape/2. - Test:
- Concurrent fan-out for the same fresh shape: assert mailbox depth is bounded by the brief-window count, and N–1 waiters receive the result via polling.
- Crash recovery: kill the GenServer mid-creation, assert lock table is empty after restart, assert new requests succeed.
- Backoff: assert poll cadence matches the configured schedule (within jitter).
- Lingua principale
- TypeScript
- Stelle
- 10.4k
- Fork
- 399
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valueForse già presa @Tyagiquamar l’ha presa 19 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Forse già presa @mohanedmoh l’ha presa 37 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4826 · 1 commento ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakeForse già presa @adityavkk l’ha presa 133 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
electric-sql/electric#4687 ·
-
Listen IP addressApertafeature request triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
electric-sql/electric#4150 · 1 commento ·
Tutte le issue di electric-sql/electric
Issue simili
-
area:cli area:security bug priority:medium
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
rtk-ai/rtk#5127 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
"code source" and "numerique.gouv" links: "new window" mention duplicated in the accessible nameApertaaccessibility
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
numerique-gouv/lasuite-landingpage#315 ·
I maintainer di solito rispondono entro 1 giorno
-
needs:triage
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 72/100
HyeokjaeChoi/vaultme#585 ·
I maintainer di solito rispondono entro 1 giorno
-
enhancement priority/P2
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
beettlle/pi-smart-router#200 ·