Query Wait Timout Error from Pgbouncer Not Propogating to Client

Open
#1,299 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
postgresql, python
Domain
databases

Research direction

Reproduce the timeout through the pgbouncer setup described in the issue, then start at asyncpg/connection.py's Connection.execute path and asyncpg/protocol/protocol.pyx query. Compare the error handling with the psycopg and psql examples, and verify that the client exposes the query_wait_timeout error rather than only reporting a closed connection.

Written by the indexing model from the issue text.

Description

When pgbouncer kills a connection due to its query_wait_timeout, the error message is not propagated to the client. All that is provided to the client is a not very helpful message "connection was closed in the middle of operation". This error message seems to be correctly handled by psql and psycopg.

I replicated this by using a pgbouncer instance with a pool_size of 1 and made sure another connection was consuming this connection to ensure that the connection would be put in a waiting mode.

asyncpg version

 > python -c 'import asyncpg; print(asyncpg.__version__)'
0.31.0

Example asyncpg code

import asyncio
import asyncpg

async def main():
    conn = await asyncpg.connect('postgresql://postgres@localhost:6432/postgres')
    row = await conn.execute('SELECT 1')

asyncio.run(main())

Stdout from the above code

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<stdin>", line 3, in main
  File "/path/to/venv/lib/python3.12/site-packages/asyncpg/connection.py", line 349, in execute
    result = await self._protocol.query(query, timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 375, in query
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation

Example code for psycopg

import psycopg
with psycopg.connect("postgresql://postgres:@localhost:6432/postgres") as conn:
    with conn.cursor() as cur:
         cur.execute("SELECT 1")

example traceback from psycopg

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/path/to/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.ProtocolViolation: query_wait_timeout

Example call and stdout from psql on query_wait_timeout

> psql \
    'postgresql://postgres:@localhost:6432/postgres' \
    --command='SELECT 1'

FATAL:  query_wait_timeout
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
connection to server was lost
Dominant language
Python
Stars
8.1k
Forks
469
Avg merge
4h 42m
Merged PRs (30d)
6

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MagicStack/asyncpg

All issues in MagicStack/asyncpg

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.