PeriodicJobEnqueuer logs ERROR "Error starting transaction: context canceled" when the client stops during the leader's staggered start

Open Beginner friendly
  • Is something wrong?
#1,576 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
66/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, postgresql
Domain
backend

Research direction

The ERROR comes from insertBatch in internal/maintenance/periodic_job_enqueuer.go, which calls s.exec.Begin(ctx) after StaggerStart returns on a cancelled context. Start by reading the stagger wait and the start-job insert path there, and check how Client.Stop cancels the maintenance services. Done means a stop during the leader's stagger logs no ERROR, while a real database failure still does.

Written by the indexing model from the issue text.

Description

### What happens

When a client that has just become the leader is stopped (SIGTERM → Client.Stop or the cancellation of the context passed to Client.Start) within the first ~0.1–2.5 s, the log gets an ERROR line even though the stop is a normal, requested shutdown:

{"level":"ERROR","msg":"maintenance.PeriodicJobEnqueuer: Error starting transaction","error":"context canceled"}
Outside that window the shutdown log is clean.

Why (as far as we can tell from the source)
The maintenance services start with StaggerStart, a cancellable sleep of a random 0–1 s per service, in sequence.
Client.Stop (and the cancellation of the start context) cancels the context of the maintenance services at once.
PeriodicJobEnqueuer wakes up from the stagger with the cancelled context and still goes on to insert the start jobs (RunOnStart periodic jobs): insertBatch calls s.exec.Begin(ctx), which fails with context.Canceled, and the error is logged at ERROR (internal/maintenance/periodic_job_enqueuer.go, insertBatch, "Error starting transaction").
Seen on v0.49.0; the same code is in v0.51.0.

Reproduction
A client with at least one PeriodicJob with RunOnStart: true, alone in the database so it becomes the leader at once.
Start it and stop it about 100 ms after it reports it is the leader (e.g. a process that gets SIGTERM right after its readiness probe).
Repeat a few times: most runs log the ERROR line above.

Expected
A requested stop does not log an error. For example, the enqueuer could return without inserting when ctx.Err() != nil after the stagger (or before Begin), or log a context cancellation during stop at a lower level, as other maintenance services do on shutdown.

Workaround we use
Our runner starts the client on a context that is not cancelled by the process shutdown and, before calling Stop, waits (bounded, a few seconds) until the leader's start jobs are in river_job or another client leads. It works, but it couples the shutdown to a database poll and delays a SIGTERM that arrives right after start.

Environment: River v0.49.0, riverpgxv5, PostgreSQL 18, Go 1.27.

Claude find it, he's always finding

Dominant language
Go
Stars
5.9k
Forks
197
Avg merge
1d 4h
Merged PRs (30d)
120

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from riverqueue/river

All issues in riverqueue/river

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.