[19.0] queue_job: Jobs stay pending after the jobrunner's database connection is silently dropped

Open
  • Is something wrong?
#984 1 comment 1 reaction 0 assignees View on GitHub

Maintainers usually reply within 1 day

@HoloborodkoBohdan is already working on this.

Since Sep 28, 2026.

  • #1000 by @HoloborodkoBohdan — open

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
postgresql, python
Domain
backend

Research direction

The issue points to queue_job/jobrunner/runner.py, specifically _connection_info_for() and QueueJobRunner.run(). Review how connection parameters are assembled and how the runner handles a lost connection; the issue proposes enabling TCP keepalives with defaults. An open linked pull request already addresses this issue, so check that work before making changes.

Written by the indexing model from the issue text.

Description

bug

Module

  • queue_job

Describe the bug

The jobrunner keeps one database connection open for as long as it runs, and it holds
the master lock on that connection.

If that connection dies without the server telling the client (a network problem, or a
database failover), two things happen:

  • PostgreSQL drops the lock, because the session is gone.
  • The runner does not notice. Its socket still looks open, so it just waits forever.

From then on no jobs are dispatched at all. Nothing is logged. The process looks healthy.
Only restarting Odoo fixes it.

The runner already has code to reconnect if the database connection breaks (the
except Exception block in run()), but it never runs, because nothing raises an error.
The runner is simply stuck waiting.

To Reproduce

Affected versions: 19.0

Steps to reproduce the behavior:

  1. Start Odoo with queue_job in server_wide_modules (odoo.conf). Check the runner has the lock:
    SELECT count(*) FROM pg_locks WHERE locktype = 'advisory'; should return 1.
  2. Break the connection silently, so the client is never told. For example, add a firewall
    rule that drops traffic between Odoo and PostgreSQL. (Using pg_terminate_backend will
    not work, because that tells the client.)
  3. Add a job. It stays pending forever. The lock count is now 0, and nothing appears in
    the log.

Expected behavior
The runner should notice the connection is dead and reconnect, which is what the
except Exception block is already there to do.

zooming in to the except Exception mentioned:
QueueJobRunner.run() in queue_job/jobrunner/runner.py.
The outer loop wraps initialize_databases() and the processing loop in a try/except,
and the last of its four handlers is:

except Exception:
    _logger.exception("exception: sleeping %ds and retrying", ERROR_RECOVERY_DELAY)
    self.close_databases()
    time.sleep(ERROR_RECOVERY_DELAY)

Additional context

  • Running Odoo in AWS ECS and Aurora PostgreSQL.
  • Odoo WEB service and CRON service separated
  • Both web and cron service has server_wide_modules = queue_job
  • cron service has http_enable = False and runs on a different port

Proposed Fix:
In queue_job/jobrunner/runner.py, at the end of
_connection_info_for(), just before the return:

# added
connection_info.setdefault("keepalives", 1)
connection_info.setdefault("keepalives_idle", 60)
connection_info.setdefault("keepalives_interval", 10)
connection_info.setdefault("keepalives_count", 5)

The fix is to turn on TCP keepalives there, so a dead connection is noticed after about a
minute instead of never.

Dominant language
Python
Stars
271
Forks
602
Avg merge
8d 3h
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OCA/queue

All issues in OCA/queue

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.