world-postgres: startup recovery enqueues the entire active set on every boot — bound it with a recovery batch size
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 62/100
- issue の種類
- 機能追加
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- node.js, postgresql, typescript
調査の方向性
Read reenqueueActiveRuns() and the createRunsList pagination path in world-postgres, then check how recovery configuration is loaded and tested. Add a configurable per-boot recovery bound while preserving the current unbounded default. Done means startup enqueues no more than the configured number of active runs and leaves the rest eligible for recovery on a later boot.
索引モデルが issue の本文から書いたものです。
説明
Summary
reenqueueActiveRuns() walks every non-terminal run on world.start(), enqueuing one Graphile job each, with no bound on how many are enqueued in a single boot. When the active set is large, a restart turns into a burst of thousands of jobs, which drives the process into memory pressure and, in our case, an out-of-memory crash loop the restart cannot escape.
Recovery already pages its read (createRunsList defaults to limit ?? 20) and awaits enqueue() one run at a time — so the loop is inherently a batch loop today. The only thing missing is a way to stop after N runs instead of draining every page.
Versions: @workflow/[email protected], @workflow/[email protected], [email protected], [email protected], Node 24.
Relationship to existing issues
Not a duplicate of the three open reports on this function; all three ask for something different:
- #3119 — which runs get selected (parked runs are replayed; duplicate jobs accumulate).
- #3968 — a producer-only World that does no startup recovery at all.
- #5129 — an enqueue-only process (
queueConcurrency: 0) that also does no startup recovery.
All three are binary: recover everything, or recover nothing. This issue is about recovering all of it, a batch at a time. A predicate fix (#3119) does not help here — the runs we hit were genuinely running with real progress records, so any correct runnable predicate still selects them. A recovery-off switch (#3968 / #5129) is also wrong for us: we want recovery, we just cannot afford 1,500 jobs in one boot.
What happens
Single-pod deployment, 4 GiB limit, FAT environment. A load test left 1,512 runs in status='running' plus a job backlog. Each restart then:
world.start()→reenqueueActiveRuns()→ lists all 1,512, enqueues 1,512 jobs.- The process works through them; RSS climbs to ~
2.8 GiBwithin a few minutes. - At the old 1 GiB limit this became an OOM crash loop: process dies → pm2 restarts → recovery re-enqueues everything again → dies again.
SIGKILLrepeated ~58 times in 30 minutes, CPU pinned at ~200% while nothing user-visible was happening.
Raising the limit to 4 GiB stopped the crash, but each boot still pushes RSS from ~600 MiB to ~2.8 GiB before settling. On a smaller instance, or with a larger active set, the same shape returns.
The amplification is in the interaction with per-job retry budgets: every recovered run gets a fresh max_attempts (MAX_GRAPHILE_JOB_ATTEMPTS = 49 + 24 in queue.ts), so a run that fails deterministically never exhausts its budget and never leaves the active set. Each restart therefore re-creates the same burst rather than draining it. (Details of that in #3119 — mentioned only because it is what makes the active set stay large.)
Reproduction
- Insert N runs with
status='running'inworkflow.workflow_runs(any N well above a few hundred reproduces the burst). - Start the World once and watch
graphile_worker._private_jobs: it gains N jobs. - Watch RSS across the first few minutes of the boot.
We observed N = 1,512 → ~1,790 jobs and RSS 600 MiB → 2.8 GiB.
Proposal
Bound the number of runs recovered per boot. Two shapes, either would work:
a) A batch size option — recover at most recoveryBatchSize runs, continue from the same cursor on the next boot:
await reenqueueActiveRuns(runs, queue.queue, 'world-postgres', namespace, {
limit: config.recoveryBatchSize, // default: unbounded (current behavior)
// the remaining runs stay non-terminal and are picked up next start
});
Because the read is already paginated and enqueue() is already awaited per run, this is a small change inside the existing loop — stop when the counter reaches the limit, leave the cursor implicit (the next boot re-reads from the beginning).
b) Page-level pacing — process at most one page per boot, or pause between pages:
for (let page = 0; hasMore; page += 1) {
if (page >= maxPagesPerBoot) break;
...
}
Either way the important property is: a bounded boot may leave runs un-recovered, and that is fine — they are still pending/running, so the next start() sees them again. Nothing is lost; the burst is spread across boots instead of landing on one.
A bounded recovery also composes with the existing per-job retry accounting: runs that do not get recovered in a given boot keep accumulating nothing, which (together with a durable enqueue identity, see #3119) would let the retry ceiling do its job.
Environment variable suggestion
Following the package's existing convention:
WORKFLOW_POSTGRES_RECOVERY_BATCH_SIZE (default: 0 = unbounded, current behavior)
Defaulting to unbounded keeps the change backwards compatible.
Current workaround
We sweep workflow_runs before eve start and flip runs that have been replayed too many times to failed, so they stop being selected. That caps the eventual damage but does not bound a single boot: while the counter is below the threshold, every boot still recovers the whole active set at once. We also tried deferring a subset without persisting any marker — that does nothing, because world.start() selects purely on status, so the "deferred" runs are recovered anyway. A recovery limit is the only way to bound the burst from outside the package, and it needs to live inside reenqueueActiveRuns.
Happy to test a branch against our environment if that helps.
- 主要言語
- TypeScript
- スター
- 2.5k
- フォーク
- 401
- 平均マージ
- 1日 21時間
- マージ済み PR(30日)
- 201
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドなし
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
vercel/workflow のほかの issue
-
getPortLazy re-runs OS port discovery on every call when it finds no port対応中かも このイシューにリンクされたプルリクエストがオープン中、またはマージ済みです。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
メンテナーはふだん 1 日以内に返信
-
node-module-esbuild-plugin: getImportedIdentifier only checks the first named import when locating a violation対応中かも @div-cowboy が 5 日前に担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
vercel/workflow#5057 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 84/100
メンテナーはふだん 1 日以内に返信
-
code-transform.mdx §ID Generation omits the package-shipped `name@version` ID form対応中かも @Mohith26 が 113 日前に担当しました。 オープン
難易度 1/5 1時間未満 初心者へのやさしさ 90/100
メンテナーはふだん 1 日以内に返信
vercel/workflow の issue をすべて見る
似ている issue
-
難易度 1/5 1〜3時間 初心者へのやさしさ 87/100
メンテナーはふだん 1 日以内に返信
-
🛠️ dette technique
難易度 2/5 1〜3時間 初心者へのやさしさ 61/100
betagouv/mon-entreprise#4752 · コメント 1 件 ·
メンテナーはふだん 5 日以内に返信
-
type: bug
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
franciscoBSalgueiro/en-croissant#1111 ·
メンテナーはふだん 2 日以内に返信
-
area/providers type/feature version/v2
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
anomalyco/opencode#64845 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
enhancement
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信