cluster: a runner that lost its shard lock can still write replies and messages for that shard

Aperta
  • Qualcosa non va?
#10,058 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
postgresql, typescript
Ambito
databases, distributed-systems

Direzione di ricerca

Start with SqlMessageStorage and SqlRunnerStorage in effect/cluster, then read the shard reset and write paths named in the issue: resetShards, saveReply, saveRequest, saveEnvelope, clearReplies and clearAddress. Run the linked repro against the current code to confirm stale writes after lock loss. Done means shard writes from an old owner are rejected after a new owner resets the shard, while client writes without a fence still work.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Base: effect 4.0.6 (effect/cluster, SqlMessageStorage + SqlRunnerStorage on PostgreSQL advisory locks). main has the same code.

What happens. Shard ownership is a session advisory lock. A runner checks it every shardLockRefreshInterval.
Between two checks, the runner can lose the lock (its lock session ended, or it was paused) while another runner
acquires the shard, runs resetShards, and starts the same entities. The old runner does not know yet. Its handlers
finish and call saveReply, saveRequest, saveEnvelope, clearReplies and clearAddress. None of these check
who owns the shard, so the writes succeed.

Results:

  • A persisted request gets two replies, or the old owner's reply wins over the new owner's.
  • A clearReplies or clearAddress from the old owner deletes what the new owner just wrote.
  • With a duplicate primary key, the write is deduplicated with no error, so nobody sees it.

Repro (two runners believe they own one shard):

const cluster = yield* make({ backend: "pg", entities: HoldEntityLayer }) // persisted RPC that waits on a gate
const [stale] = yield* cluster.start(1)
const reply = yield* client("held").Hold(new Request({ id: "fenced" })).pipe(Effect.forkChild)
// handler has started on `stale`
yield* cluster.freeze(stale)              // stale stops refreshing
yield* cluster.dropLockSessions           // pg_terminate_backend on its lock session
const [owner] = yield* cluster.start(1)   // owner acquires the shard and re-runs the request
assert(stale.sharding.hasShardId(shard))  // stale still believes it owns it
openGate(stale)                           // stale's handler finishes first
// 4.0.6: stale's reply is stored and returned to the caller.
// Expected: stale's saveReply fails; only owner's reply is stored.

Expected. Each shard acquisition gets an epoch. Every write on behalf of a shard carries the epoch the writer
acquired with, and storage rejects it in the same transaction if the shard's epoch has moved on. The rejection is a
typed error, logged at Error, and the entity on the stale runner stops.

Proposed fix (implemented on a fork, PR available).

  • SqlMessageStorage migration 0004_shard_epochs adds <prefix>_shards (shard_id, epoch).
  • resetShards raises the epoch of each shard it resets and returns a ShardFence per shard.
  • MessageStorage.CurrentShardFence carries the fence to the write. saveRequest, saveEnvelope, saveReply,
    clearReplies and clearAddress read the shard's epoch with a shared row lock (FOR SHARE, LOCK IN SHARE MODE,
    HOLDLOCK) inside the write's transaction, and fail with ShardFenced if it differs. resetShards takes the
    exclusive lock on the same row, so a stale write either commits before the new owner's reset or fails.
  • Writes by a client (sending to a shard it does not own) carry no fence and are not checked.
  • An entity whose reply is fenced is stopped. Sharding releases a shard whose reset fails.

Runnable repro (real runner processes, fails on 4.0.6 and on main at 837e906): https://github.com/heyimcarlos/effect-cluster-repro, issue 3 in the README table.

Lingua principale
TypeScript
Stelle
16.7k
Fork
808
Merge medio
11h 3m
PR unite (30g)
840

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Effect-TS/effect

Tutte le issue di Effect-TS/effect

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.