[19.0] queue_job: Jobs stay pending after the jobrunner's database connection is silently dropped
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 2/5
- Tempo stimato
- 1-3 ore
- Idoneità per principianti
- 25/100
Direzione di ricerca
The issue points to queue_job/jobrunner/runner.py, specifically _connection_info_for() and QueueJobRunner.run(). Review how connection parameters are assembled and how the runner handles a lost connection; the issue proposes enabling TCP keepalives with defaults. An open linked pull request already addresses this issue, so check that work before making changes.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Module
- queue_job
Describe the bug
The jobrunner keeps one database connection open for as long as it runs, and it holds
the master lock on that connection.
If that connection dies without the server telling the client (a network problem, or a
database failover), two things happen:
- PostgreSQL drops the lock, because the session is gone.
- The runner does not notice. Its socket still looks open, so it just waits forever.
From then on no jobs are dispatched at all. Nothing is logged. The process looks healthy.
Only restarting Odoo fixes it.
The runner already has code to reconnect if the database connection breaks (the
except Exception block in run()), but it never runs, because nothing raises an error.
The runner is simply stuck waiting.
To Reproduce
Affected versions: 19.0
Steps to reproduce the behavior:
- Start Odoo with
queue_jobinserver_wide_modules(odoo.conf). Check the runner has the lock:
SELECT count(*) FROM pg_locks WHERE locktype = 'advisory';should return 1. - Break the connection silently, so the client is never told. For example, add a firewall
rule that drops traffic between Odoo and PostgreSQL. (Usingpg_terminate_backendwill
not work, because that tells the client.) - Add a job. It stays
pendingforever. The lock count is now 0, and nothing appears in
the log.
Expected behavior
The runner should notice the connection is dead and reconnect, which is what the
except Exception block is already there to do.
zooming in to the except Exception mentioned:
QueueJobRunner.run() in queue_job/jobrunner/runner.py.
The outer loop wraps initialize_databases() and the processing loop in a try/except,
and the last of its four handlers is:
except Exception:
_logger.exception("exception: sleeping %ds and retrying", ERROR_RECOVERY_DELAY)
self.close_databases()
time.sleep(ERROR_RECOVERY_DELAY)
Additional context
- Running Odoo in AWS ECS and Aurora PostgreSQL.
- Odoo WEB service and CRON service separated
- Both web and cron service has server_wide_modules = queue_job
- cron service has http_enable = False and runs on a different port
Proposed Fix:
In queue_job/jobrunner/runner.py, at the end of
_connection_info_for(), just before the return:
# added
connection_info.setdefault("keepalives", 1)
connection_info.setdefault("keepalives_idle", 60)
connection_info.setdefault("keepalives_interval", 10)
connection_info.setdefault("keepalives_count", 5)
The fix is to turn on TCP keepalives there, so a dead connection is noticed after about a
minute instead of never.
- Lingua principale
- Python
- Stelle
- 271
- Fork
- 602
- Merge medio
- 4g 4h
- PR unite (30g)
- 6
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OCA/queue
-
enhancement
Difficoltà 3/5 1-2 giorni Idoneità per principianti 30/100
I maintainer di solito rispondono entro 1 giorno
-
Migration to version 20.0Apertahelp wanted no stale work in progress
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
Delaying unlink or write on a model whose method was replaced raises AttributeError at enqueue timeForse già presa Una pull request collegata a questa issue è aperta o già unita. Apertabug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
enhancement
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
OCA/queue#964 · 1 commento · 1 reazione ·
I maintainer di solito rispondono entro 1 giorno
Issue simili
-
getchaintips leaves out the active tip once a header is staged on it, where Core always lists itAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
btclib-org/btclib-node#2158 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
dbt-msft/dbt-sqlserver#897 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
Gathin23/cleanyourtv#2770 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
vllm-project/vllm#64845 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
dekobon/big-code-analysis#1729 ·
I maintainer di solito rispondono entro 1 giorno