Memory leak with Pool usage

Open
#1,024 3 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
postgresql, python
Domain
databases

Research direction

Start at the asyncpg.create_pool entry point with the reported pool arguments, and reproduce the gradual memory increase using the described psutil measurement. Compare pooled connections with direct asyncpg connections, including after max_inactive_connection_lifetime, explicit garbage collection, and pool closure. Done means the reproduction no longer shows query-count-related growth.

Written by the indexing model from the issue text.

Description

  • asyncpg version: 0.27
  • PostgreSQL version: 14
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?    : no
  • Python version: 3.8, 3.10 and 3.11
  • Platform: docker images python:3.x-slim-buster
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: yes
  • Can the issue be reproduced under both asyncio and
    uvloop?    : only asyncio

When we use a Pool, the memory usage of our process keeps increasing slowly with the number of queries. After 1000 queries, it is ~10MB and this increases with the number of queries executed on connections acquired from the pool. This remains after the max_inactive_connection_lifetime has passed with no activity, with explicit garbage collection and even after we close the pool. We can reproduce this with a psutil measurement and observe it in production. When we replace the usage of Pool with our own creation/cleanup of asynpg Connections, the issue dissapears.

Pool created with the following arguments:

asyncpg.create_pool(
    max_inactive_connection_lifetime=300,
    command_timeout=30,
    min_size=0,
    max_size=5
)
Dominant language
Python
Stars
8.1k
Forks
469
Avg merge
2d 20h
Merged PRs (30d)
9

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from MagicStack/asyncpg

All issues in MagicStack/asyncpg

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.