Owner-aware / single-flight boot recovery for multi-instance self-hosted deployments
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- postgresql, typescript
- Domain
- backend, databases, distributed-systems
Research direction
Start with start() and ensureWorldStarted(), then read the context in #2544, #2978, and #2780. Compare the proposed single-flight and owner-aware directions; done should prevent redundant boot recovery scans and duplicate enqueues for multi-instance deployments.
Written by the indexing model from the issue text.
Description
Follow-up from #2544 (boot-time recovery via ensureWorldStarted()).
Problem
start()'s boot recovery re-enqueues every pending/running run in storage — it does not know which instance is responsible for a run. On a horizontally-scaled or rolling deployment of the Postgres World, every booting instance therefore performs a full paginated scan of the active runs and re-enqueues all of them: N replicas ⇒ N scans + N duplicate enqueues per in-flight run on every deploy. Replay idempotency, the inline-ownership lease, and job-key dedup make this correct, but at scale it is a thundering herd of redundant scans, enqueues, and wake replays.
Current mitigation
WORKFLOW_SKIP_BOOT_RECOVERY=1 (added in #2544) lets operators run recovery from a single designated instance and opt the rest out. That is manual designation, not a mechanism.
Possible directions
- Single-flight: an advisory lock (e.g.
pg_advisory_lock) or a recovery-epoch row so only one booting instance per deploy performs the scan. - Owner-aware: attribute runs to instances (lease/heartbeat) so a booting instance only re-drives runs whose owner is gone; this would also shorten the mid-step recovery delay currently bounded by the inline-ownership lease (
WORKFLOW_INLINE_OWNERSHIP_LEASE_SECONDS, default 860s), since a dead owner could be detected affirmatively instead of waiting out the lease.
Related: #2978 (shared-DB namespace isolation for recovery), #2780 (inline-ownership lease).
🤖 Generated with Claude Code
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 365
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 160
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vercel/workflow
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug clawsweeper:linked-pr-open clawsweeper:needs-live-repro clawsweeper:no-new-fix-pr impact:message-loss issue-rating: 🐚 platinum hermit P2 regression
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
fullcalendar/fullcalendar#8106 ·