world-postgres 4.3.x (latest): close() can resolve while a job is still running — fixed on main, not on the 4.x line

Bắt đầu trong dist/queue.js và so sánh đường dẫn shutdown của 4.3.x với phần triển khai tương ứng trên main. Chạy bản tái hiện Graphile-layer được cung cấp bằng một DATABASE_URL có role sở hữu CREATEDB, sau đó xác minh rằng shutdown vẫn ở trạng thái chờ cho đến khi job đang hoạt động hoàn tất và hàng của nó được xóa.

Summary

On the latest release line, PostgresWorld.close() can resolve while a step job is still running, so the application tears down its pool (and exits) underneath a job that has not yet acknowledged completion. The cause is the interaction between Graphile's own signal handlers and runner.stop().

This is already fixed on main. Both fixes are present in the published 5.0.0-beta.44 and absent from 4.3.7, which is what latest currently points at. The ask is a backport to the 4.3.x line, since 5.0 is still beta and moving the whole SDK ([email protected] → 5.0.0-beta.53) is a much larger step than the shutdown fix warrants.

The race on 4.3.7

dist/queue.js calls Graphile without noHandleSignals, so Graphile installs its own SIGTERM/SIGINT handlers:

runner = await run({
    pgPool: pool,
    concurrency: config.queueConcurrency || 10,
    logger: graphileLogger,
    pollInterval: 500,
    taskList,
});

and close() does not await the pool after stopping the runner:

async close() {
    closing = true;
    // ...
    if (runner) {
        await runner.stop();
        runner = null;
    }
    if (workerUtils) {
        await workerUtils.release();
        workerUtils = null;
    }
    // ...
}

In [email protected] (the pinned version), stop() only awaits the worker pool when it is still active:

const promises = [];
if (cron._active) {
    promises.push(cron.release());
}
if (workerPool._active) {
    promises.push(workerPool.gracefulShutdown());
}
await Promise.all(promises).then(release);

So on SIGTERM the ordering is:

1. Graphile's own handler starts workerPool.gracefulShutdown(), which clears _active before in-flight jobs finish.
2. The application calls world.close() → runner.stop().
3. _active is already false, so stop() awaits nothing, runs release() and resolves.
4. close() continues: workerUtils.release(), pool teardown, process exit — while a job still needs a connection to persist its result.

The window is small but real, and it is exactly the case an application-managed shutdown is supposed to cover: the whole point of awaiting world.close() is that it means "jobs are done".

Reproduction

At the Graphile layer, with the same options world-postgres passes. Needs a DATABASE_URL whose role has CREATEDB; it creates and drops a disposable database. On [email protected] this prints the failure; with either upstream fix applied it prints PASS.

const { Pool } = require("pg");
const { run } = require("graphile-worker");

const verify = async () => {
  const connectionString = process.env.DATABASE_URL;
  if (!connectionString) throw new Error("DATABASE_URL with CREATEDB is required");

  const databaseName = `gw_shutdown_check_${process.pid}_${Date.now()}`;
  const admin = new Pool({ connectionString, connectionTimeoutMillis: 3000 });
  const entered = Promise.withResolvers();
  const release = Promise.withResolvers();
  const originalKill = process.kill;
  let taskPool, runner, stopping, created = false, jobFinished = false;

  try {
    await admin.query(`CREATE DATABASE "${databaseName}"`);
    created = true;
    const databaseUrl = new URL(connectionString);
    databaseUrl.pathname = `/${databaseName}`;
    taskPool = new Pool({ connectionString: databaseUrl.toString(), connectionTimeoutMillis: 3000 });

    runner = await run({
      concurrency: 1,
      pgPool: taskPool,
      pollInterval: 50,
      taskList: {
        hold: async () => {
          entered.resolve();
          await release.promise;
          jobFinished = true;
        },
      },
    });
    await runner.addJob("hold", {});
    await entered.promise;

    // Deliver the signal Graphile is listening for, but keep this process alive.
    process.kill = (pid, signal) => (pid === process.pid ? true : originalKill(pid, signal));
    process.emit("SIGTERM", "SIGTERM");

    let stopResolved = false;
    stopping = runner.stop().then(() => { stopResolved = true; });
    await new Promise((done) => setTimeout(done, 100));

    if (stopResolved || jobFinished) {
      throw new Error("Runner stopped before its active job finished");
    }

    release.resolve();
    await stopping;
    await runner.promise;
    const result = await taskPool.query("SELECT count(*)::int AS count FROM graphile_worker.jobs");
    if (!jobFinished || result.rows[0].count !== 0) {
      throw new Error("Job completion was not acknowledged before stop resolved");
    }
    console.log("PASS: shutdown waited for the job and its acknowledgement");
  } finally {
    release.resolve();
    if (runner) { await (stopping ?? runner.stop()); await runner.promise; }
    await taskPool?.end();
    if (created) await admin.query(`DROP DATABASE "${databaseName}"`);
    await admin.end();
    process.kill = originalKill;
  }
};

verify().catch((error) => { console.error(error.message); process.exitCode = 1; });

Expected: the runner stays pending until the job completes and its row is gone.
Actual on 4.3.7's dependency set: Runner stopped before its active job finished.

The fix that already exists on main

5.0.0-beta.44, dist/queue.js:

runner = await run({
    pgPool: pool,
    concurrency: config.queueConcurrency || 50,
    logger: graphileLogger,
    ...(config.applicationManagedShutdown === true && {
        noHandleSignals: true,
    }),
    pollInterval: 500,
    taskList,
});

const activeRunner = runner;
if (activeRunner) {
    try {
        await activeRunner.stop();
    } catch (error) { /* "Runner is already stopped" */ }
    await activeRunner.promise.catch(() => {});
    runner = null;
}

Either change alone closes the race. Note that 5.0.0-beta.44 still pins [email protected], so this is a World-level fix and not something a Graphile bump would deliver.

Request

Backport one or both to 4.3.x. await runner.promise after runner.stop() is the smaller of the two and is safe regardless of who owns the signal, so it would be enough on its own.

If a backport is not planned, it would help to say so — then the answer for self-hosted 4.x users is "pin a patch until 5.0 is stable", which is what we are doing (a three-line patch adding promises.push(workerPool.promise) to Graphile's stop()), and knowing that is the intended state is useful.

Environment

• @workflow/[email protected] (latest), [email protected], [email protected]
• Next.js 16 App Router, Bun 1.4.2 (bun --bun next start), self-hosted on Kubernetes
• Application-managed shutdown: the HTTP server owns SIGTERM and drains, then awaits world.close() before exiting