world-postgres: support an enqueue-only process (`WORKFLOW_POSTGRES_WORKER_CONCURRENCY=0` / `queueConcurrency: 0`)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- node.js, postgresql, typescript
Research direction
Start by reading the Postgres World's createWorld(), startRunnerWhenExecutorIsReady(), setupListeners(), and queue() paths in dist/index.js and dist/queue.js; compare them with the source and current tests. Check how zero is parsed and passed to Graphile's run(), and how the loopback deferred start behaves. Done means a zero-concurrency World can enqueue after migrations without starting a runner, while a nonzero concurrency still runs jobs; the issue offers a real Postgres test scenario.
Written by the indexing model from the issue text.
Description
What we want
We run the Postgres World in two kinds of processes of the same Next app:
- web replicas that serve HTTP and start runs (
start()→world.queue()), but should never execute a flow or step; - worker replicas that run Graphile's runner and execute everything over loopback.
Separating them keeps long steps out of the web replicas (no job slots held there, no drain on shutdown) and lets the two scale independently. For this the web replicas need a World whose queue() adds the Graphile job but whose start() never starts a runner.
What happens today (@workflow/[email protected], and the same on main)
There is no enqueue-only mode, and a concurrency of 0 cannot express one:
createWorld()'s default config computesqueueConcurrency: parseInt(process.env.WORKFLOW_POSTGRES_WORKER_CONCURRENCY || '10', 10) || 10(|| '50' … || 50onmain), soWORKFLOW_POSTGRES_WORKER_CONCURRENCY=0becomes 10 (50).setupListeners()passesconcurrency: config.queueConcurrency || 10(|| 50onmain) to Graphile'srun(), so an explicitqueueConcurrency: 0also becomes 10 (50).queue()awaitsstart(), andstart()always callsstartRunnerWhenExecutorIsReady()aftermakeWorkerUtils()/migrate(), starting the runner at once or deferred behind the loopback probe. So any process that enqueues also executes.
Proposal
Treat a concurrency of 0 as "enqueue only":
- keep a parsed
0in the default config (fall back to the default only when the variable is missing or not a number); - in
startRunnerWhenExecutorIsReady(), return early whenconfig.queueConcurrency === 0(this covers both call sites instart()and the deferred loopback path), whilestart()still makes the worker utils and runs the Graphile and pg-boss migrations so thatqueue()works; - use
config.queueConcurrency ?? <default>insetupListeners().
close() already handles a missing runner, and reenqueueActiveRuns in world.start() stays harmless on such a process since it enqueues by job key. An explicit option (e.g. mode: 'enqueue-only' or runner: false) would be equally fine for us if you prefer it over overloading 0; a 4.x release with it would let us drop our patch.
What we do meanwhile
We pin a three-place patch of dist/index.js and dist/queue.js of 4.3.7 (via Bun's patchedDependencies) doing exactly the above, and verify it against real Postgres: a World with queueConcurrency: 0 (explicitly and through WORKFLOW_POSTGRES_WORKER_CONCURRENCY=0) enqueues a workflow message that stays in graphile_worker.jobs while a stub executor receives no request; a World with 10 delivers it and the job is gone. Happy to open a PR with the change and such a test if that helps.
Related, noticed while testing (separate issue if you like)
Under Bun, world.close() throws TypeError: httpAgent?.close is not a function from the embedded local World (@workflow/[email protected], dist/queue.js close()), apparently because the undici Agent that Bun resolves there has no close(); the Postgres World's close() then never reaches pool.end(). Setting WORKFLOW_NODE_HTTP=1 avoids it.
- Dominant language
- TypeScript
- Stars
- 2.5k
- Forks
- 401
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 201
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vercel/workflow
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 1 day
-
node-module-esbuild-plugin: getImportedIdentifier only checks the first named import when locating a violationPossibly taken @div-cowboy claimed this 5 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
vercel/workflow#5057 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
code-transform.mdx §ID Generation omits the package-shipped `name@version` ID formPossibly taken @Mohith26 claimed this 113 days ago. Open
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
Similar issues
-
db-d1-sqlite: upsert is aliased to updateOne, so payload.kv.set never inserts new keysPossibly taken @pcbeingused333 claimed this today. OpenBug db: d1-sqlite Payload OSS
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
payloadcms/payload#19766 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
device-management-toolkit/sample-web-ui#3711 ·
Maintainers usually reply within 1 day
-
bug impact:low Team: SecuritySolution Team:Defend Workflows v9.6.0
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
elastic/kibana#306401 · 3 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
agentes-chat/agentes-chat#927 ·
Maintainers usually reply within 1 day
-
Closing a pinned tab jumps to the Tabs section instead of the next open pinned tabPossibly taken @seenimohamed claimed this today. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ardia-browser/ardia#2769 ·
Maintainers usually reply within 1 day