world-postgres: support an enqueue-only process (`WORKFLOW_POSTGRES_WORKER_CONCURRENCY=0` / `queueConcurrency: 0`)

Đang mở
#5,129 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
68/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
node.js, postgresql, typescript
Lĩnh vực
backend, databases

Hướng nghiên cứu

Start by reading the Postgres World's createWorld(), startRunnerWhenExecutorIsReady(), setupListeners(), and queue() paths in dist/index.js and dist/queue.js; compare them with the source and current tests. Check how zero is parsed and passed to Graphile's run(), and how the loopback deferred start behaves. Done means a zero-concurrency World can enqueue after migrations without starting a runner, while a nonzero concurrency still runs jobs; the issue offers a real Postgres test scenario.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

What we want

We run the Postgres World in two kinds of processes of the same Next app:

  • web replicas that serve HTTP and start runs (start() → world.queue()), but should never execute a flow or step;
  • worker replicas that run Graphile's runner and execute everything over loopback.

Separating them keeps long steps out of the web replicas (no job slots held there, no drain on shutdown) and lets the two scale independently. For this the web replicas need a World whose queue() adds the Graphile job but whose start() never starts a runner.

What happens today (@workflow/[email protected], and the same on main)

There is no enqueue-only mode, and a concurrency of 0 cannot express one:

  1. createWorld()'s default config computes queueConcurrency: parseInt(process.env.WORKFLOW_POSTGRES_WORKER_CONCURRENCY || '10', 10) || 10 (|| '50' … || 50 on main), so WORKFLOW_POSTGRES_WORKER_CONCURRENCY=0 becomes 10 (50).
  2. setupListeners() passes concurrency: config.queueConcurrency || 10 (|| 50 on main) to Graphile's run(), so an explicit queueConcurrency: 0 also becomes 10 (50).
  3. queue() awaits start(), and start() always calls startRunnerWhenExecutorIsReady() after makeWorkerUtils() / migrate(), starting the runner at once or deferred behind the loopback probe. So any process that enqueues also executes.
Proposal

Treat a concurrency of 0 as "enqueue only":

  • keep a parsed 0 in the default config (fall back to the default only when the variable is missing or not a number);
  • in startRunnerWhenExecutorIsReady(), return early when config.queueConcurrency === 0 (this covers both call sites in start() and the deferred loopback path), while start() still makes the worker utils and runs the Graphile and pg-boss migrations so that queue() works;
  • use config.queueConcurrency ?? <default> in setupListeners().

close() already handles a missing runner, and reenqueueActiveRuns in world.start() stays harmless on such a process since it enqueues by job key. An explicit option (e.g. mode: 'enqueue-only' or runner: false) would be equally fine for us if you prefer it over overloading 0; a 4.x release with it would let us drop our patch.

What we do meanwhile

We pin a three-place patch of dist/index.js and dist/queue.js of 4.3.7 (via Bun's patchedDependencies) doing exactly the above, and verify it against real Postgres: a World with queueConcurrency: 0 (explicitly and through WORKFLOW_POSTGRES_WORKER_CONCURRENCY=0) enqueues a workflow message that stays in graphile_worker.jobs while a stub executor receives no request; a World with 10 delivers it and the job is gone. Happy to open a PR with the change and such a test if that helps.

Related, noticed while testing (separate issue if you like)

Under Bun, world.close() throws TypeError: httpAgent?.close is not a function from the embedded local World (@workflow/[email protected], dist/queue.js close()), apparently because the undici Agent that Bun resolves there has no close(); the Postgres World's close() then never reaches pool.end(). Setting WORKFLOW_NODE_HTTP=1 avoids it.

Ngôn ngữ chính
TypeScript
Star
2.5k
Fork
401
Merge trung bình
1 ngày 21 giờ
Pull request đã merge (30 ngày)
201

Chuẩn bị môi trường

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của vercel/workflow

Tất cả issue của vercel/workflow

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.