cluster: a runner that lost its shard lock can still write replies and messages for that shard

未关闭
  • 哪里不对？
#10,058 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
postgresql, typescript
领域
databases, distributed-systems

调研方向

Start with SqlMessageStorage and SqlRunnerStorage in effect/cluster, then read the shard reset and write paths named in the issue: resetShards, saveReply, saveRequest, saveEnvelope, clearReplies and clearAddress. Run the linked repro against the current code to confirm stale writes after lock loss. Done means shard writes from an old owner are rejected after a new owner resets the shard, while client writes without a fence still work.

由索引模型根据 Issue 内容生成。

描述

Base: effect 4.0.6 (effect/cluster, SqlMessageStorage + SqlRunnerStorage on PostgreSQL advisory locks). main has the same code.

What happens. Shard ownership is a session advisory lock. A runner checks it every shardLockRefreshInterval.
Between two checks, the runner can lose the lock (its lock session ended, or it was paused) while another runner
acquires the shard, runs resetShards, and starts the same entities. The old runner does not know yet. Its handlers
finish and call saveReply, saveRequest, saveEnvelope, clearReplies and clearAddress. None of these check
who owns the shard, so the writes succeed.

Results:

  • A persisted request gets two replies, or the old owner's reply wins over the new owner's.
  • A clearReplies or clearAddress from the old owner deletes what the new owner just wrote.
  • With a duplicate primary key, the write is deduplicated with no error, so nobody sees it.

Repro (two runners believe they own one shard):

const cluster = yield* make({ backend: "pg", entities: HoldEntityLayer }) // persisted RPC that waits on a gate
const [stale] = yield* cluster.start(1)
const reply = yield* client("held").Hold(new Request({ id: "fenced" })).pipe(Effect.forkChild)
// handler has started on `stale`
yield* cluster.freeze(stale)              // stale stops refreshing
yield* cluster.dropLockSessions           // pg_terminate_backend on its lock session
const [owner] = yield* cluster.start(1)   // owner acquires the shard and re-runs the request
assert(stale.sharding.hasShardId(shard))  // stale still believes it owns it
openGate(stale)                           // stale's handler finishes first
// 4.0.6: stale's reply is stored and returned to the caller.
// Expected: stale's saveReply fails; only owner's reply is stored.

Expected. Each shard acquisition gets an epoch. Every write on behalf of a shard carries the epoch the writer
acquired with, and storage rejects it in the same transaction if the shard's epoch has moved on. The rejection is a
typed error, logged at Error, and the entity on the stale runner stops.

Proposed fix (implemented on a fork, PR available).

  • SqlMessageStorage migration 0004_shard_epochs adds <prefix>_shards (shard_id, epoch).
  • resetShards raises the epoch of each shard it resets and returns a ShardFence per shard.
  • MessageStorage.CurrentShardFence carries the fence to the write. saveRequest, saveEnvelope, saveReply,
    clearReplies and clearAddress read the shard's epoch with a shared row lock (FOR SHARE, LOCK IN SHARE MODE,
    HOLDLOCK) inside the write's transaction, and fail with ShardFenced if it differs. resetShards takes the
    exclusive lock on the same row, so a stale write either commits before the new owner's reset or fails.
  • Writes by a client (sending to a shard it does not own) carry no fence and are not checked.
  • An entity whose reply is fenced is stopped. Sharding releases a shard whose reset fails.

Runnable repro (real runner processes, fails on 4.0.6 and on main at 837e906): https://github.com/heyimcarlos/effect-cluster-repro, issue 3 in the README table.

主要语言
TypeScript
星标
16.7k
派生
808
平均合并
10 小时 15 分钟
30 天内合并 PR
845

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

Effect-TS/effect 的其他 Issue

查看 Effect-TS/effect 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。