`runner.promise` resolves when the worker pool exits with an error

Start in src/runner.ts at buildRunner and inspect how workerPool.promise and cron.promise feed into the runner promise. Add the reproduction test described in the issue, then run the relevant test suite. Done means a worker-pool error rejects runner.promise with the original error and the CLI exits with code 1.

Summary

Since 0.17.0, when the worker pool shuts itself down because of an error (for example Could not completeJobs; queue is in an inconsistent state; aborting.), runner.promise resolves, the same as after runner.stop(). The error only appears in the log line Runner stopping (reason: worker pool exited with error: …).

A host that treats a rejected runner.promise as the failure signal never finds out, and its process keeps running without a worker pool. The CLI is affected too: it awaits the same promise, so a pool crash now ends the CLI with exit code 0.

Steps to reproduce

Make one job's completion fail, then watch how runner.promise settles:

test("runner.promise rejects when the worker pool exits with an error", () =>
  withOptions(async (options) => {
    const { pgPool } = options;
    await reset(pgPool, options);
    await pgPool.query(`
      create or replace function public.refuse_job_completion() returns trigger
      language plpgsql as $$ begin raise exception 'completion refused'; end; $$;
      create trigger refuse_job_completion
        before delete on ${ESCAPED_GRAPHILE_WORKER_SCHEMA}._private_jobs
        for each row execute function public.refuse_job_completion();
    `);

    const runner = await run({
      ...options,
      taskList: { job1: async () => {} },
      preset: { worker: { completeJobBatchDelay: 0 } },
    });
    try {
      await runner.addJob("job1", { id: "1" });
      const outcome = await runner.promise.then(
        () => "resolved",
        (error: unknown) => `rejected: ${String(error)}`,
      );
      expect(outcome).toMatch(/^rejected: .*completion refused/);
    } finally {
      await runner.stop().catch(() => {});
    }
  }));

(completeJobBatchDelay: 0 only makes the pool give up immediately; with unbatched completions it retries with backoff first and ends the same way.)

On main (4cda192) this fails:

[core] ERROR: Failed to complete jobs '1':
[core] WARNING: Runner stopping (reason: worker pool exited with error: error: completion refused)
Expected pattern: /^rejected: .*completion refused/
Received string:  "resolved"

Expected

runner.promise rejects with the pool's error, as the error branch in buildRunner is written to do (logger.error("Stopping worker due to an error: …") then return Promise.reject(error)).

The library example in the configuration docs relies on this too: await runner.promise inside main(), with main().catch(… process.exit(1)). With the current behaviour that example exits with code 0 after the pool fails. (pool:fatalError is emitted, but it isn't listed in the worker events docs.)

Cause

src/runner.ts, buildRunner: the pool's (and cron's) rejection is turned into a stop(…) call and then swallowed:

const wp = workerPool.promise
  .then(
    () => (running ? stop("worker pool exited cleanly", true) : void 0),
    (e) => (running ? stop(`worker pool exited with error: ${e}`) : void 0),
  )
  .catch(noop);

so Promise.all([cp, wp]) always resolves and its rejection handler can't run for a pool or cron failure.

This came in with 6b63ebd ("Massively improve logging around worker shutdown", part of #544), before 0.17.0 shipped. Running the test above on different versions:

So the rework's intended rejection was lost in the same change that added the new shutdown logging.

Possible fix

Let the rejection reach the existing handler, which already stops the runner and rejects:

const wp = workerPool.promise.then(() =>
  running ? stop("worker pool exited cleanly", true) : void 0,
);
const cp = cron.promise.then(() =>
  running ? stop("cron exited cleanly", true) : void 0,
);

With this the test above passes and the rest of the suite is unchanged locally. In the logs, the runner now also prints Stopping worker due to an error: …, and its stop reason reads error: … instead of worker pool exited with error: ….

The CLI shows the same difference. With a job whose completion fails (same trigger, completeJobBatchDelay: 0 in the config), graphile-worker built from main logs the failure and exits with code 0; built with the fix it logs Stopping worker due to an error: error: completion refused and exits with code 1.

Happy to open a PR with the test and fix, or adjust to keep the old wording.

Additional context

• graphile-worker 0.18.0 (also checked with @graphile-pro/worker 0.2.2 loaded: same behaviour), Node 24 and Bun, PostgreSQL 16.
• Related: #501 (process not exiting after seppuku, so the supervisor never restarts it) and #592 (0.16.6: runner.promise never settled). The 0.17 rework fixed the "never settles" case from #592; this is the remaining half: it settles, but as a success.