in_transaction() leaks a pool connection when begin() is cancelled

Open
  • Is something wrong?
#2,391 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

@inchang-ing is already working on this.

Since Nov 1, 2026.

  • #2392 by @inchang-ing — open

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
postgresql, python
Domain
databases

Research direction

Start in TransactionContextPooled.__aenter__, where the connection is acquired from the pool before begin() runs outside any cleanup. Read __aexit__ to see how the release is handled, and check how the asyncpg pool's release is shielded from cancellation. Done means a cancelled begin() returns the connection, which the issue's reproduction script can confirm. Open PR #2293 edits the same lines, so check it before starting.

Written by the indexing model from the issue text.

Description

Describe the bug

TransactionContextPooled.__aenter__ takes a connection from the pool, sets the context variable, then awaits begin() with nothing around it. If the task is cancelled during begin(), for example by a request deadline, __aenter__ never finishes, so __aexit__ never runs and the connection is never given back. Each time, the pool loses one connection for good. With no acquire timeout set, once they are all gone every later query waits on pool.acquire(). In my runs the next request timed out after 5 s.

in_transaction() is also used inside the ORM (for example get_or_create, as in #1152), so code that never opens a transaction itself can hit this. #1152 mentions the same thing happening in __aenter__; the __aexit__ part of that issue is covered now by the try/finally.

To Reproduce

Postgres 16.2, asyncpg 0.32.0, anyio 4.16.0, Python 3.12, tortoise on develop at cdef122 (the class is the same in 1.1.9). This puts a 100 ms anyio deadline around an ordinary 20 ms transaction, with more requests than the pool has connections:

import asyncio

import anyio
from tortoise import Tortoise, connections, fields
from tortoise.models import Model
from tortoise.transactions import in_transaction

DB_URL = "postgres://[email protected]:5532/postgres?minsize=1&maxsize=10"


class Item(Model):
    id = fields.IntField(primary_key=True)


async def handler():
    async with in_transaction() as conn:
        await conn.execute_query("SELECT pg_sleep(0.02)")


async def request():
    # a 100 ms deadline around an ordinary 20 ms transaction
    with anyio.move_on_after(0.1) as scope:
        await handler()
    return scope.cancelled_caught


async def main():
    await Tortoise.init(db_url=DB_URL, modules={"models": ["__main__"]})
    await handler()
    timeouts = 0
    for _ in range(20):  # 20 waves of 150 concurrent requests
        results = await asyncio.gather(*[asyncio.create_task(request()) for _ in range(150)])
        timeouts += sum(results)
    await asyncio.sleep(0.3)
    pool = connections.get("default")._pool
    print(f"timeouts: {timeouts}, pool size: {pool.get_size()}, idle: {pool.get_idle_size()}")
    try:
        async with asyncio.timeout(5):
            await handler()
        print("next request: ok")
    except TimeoutError:
        print("next request: timed out after 5 s waiting for a connection")
    await Tortoise.close_connections()


asyncio.run(main())

One run on develop:

timeouts: 2413, pool size: 10, idle: 0
next request: timed out after 5 s waiting for a connection

The counts vary from run to run. Over 8 runs the leaked connections were 10, 9, 9, 9, 9, 7, 5 and 0. In 3 of those runs I wrapped begin() to count the CancelledErrors coming out of it, and the count matched the leaked connections each time (5, 9 and 9).

A plain asyncio.timeout() leaks the same way when the deadline lands inside BEGIN. With 1000 requests one after another, a pool of 50 and random deadlines of 0-5 ms, 16 connections leaked.

Expected behavior

A request that is cancelled while its transaction is starting gives the connection back, the same as one cancelled inside the block.

With this change to TransactionContextPooled.__aenter__, the same script leaked 0 connections in 6 runs out of 6, and the next request was ok each time:

    async def __aenter__(self) -> TransactionalDBClient:
        await self.ensure_connection()
        self.client._connection = await self.client._parent._pool.acquire()
        self.token = get_connections().set(self.connection_name, self.client)
        try:
            await self.client.begin()
        except BaseException:
            get_connections().reset(self.token)
            await self.client._parent._pool.release(self.client._connection)
            raise
        return self.client

I have only measured this on asyncpg, where Pool.release() is shielded, so the release finishes while the task is being cancelled. The psycopg backend releases with a plain await self.putconn(connection), and I have not checked mysql, mssql or oracle, so those may also need a shield or some handling of a failed release.

Additional context

With the change, asyncpg logs Resetting connection with an active transaction for some of the cancelled requests, because BEGIN had already reached the server before the cancel. The same warning is the subject of #1094; I don't know whether it has the same cause there.

Possibly related, different trigger: #1438. Open PR #2293 edits these same lines; I can wait for it or rebase on it.

I can open a PR with this change and a test if you want one.

Dominant language
Python
Stars
5.6k
Forks
540
Avg merge
4d 10h
Merged PRs (30d)
31

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tortoise/tortoise-orm

All issues in tortoise/tortoise-orm

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.