BUG: Pool not closing pipes after close and join

Open
  • Is something wrong?
#282 16 comments 0 reactions 7 assignees View on GitHub

@gautamp8 is already working on this.

Since Feb 28, 2021.

Assessment

This issue has not been assessed yet.

Description

Issue Type: Bug

I'm using billiard.Pool to run parallel computations.
I have a continuous service (celery) doing it and after a while I get an OSError 24 ("Too many open files").
While debugging I found that a Pool creates a number of pipes:
in pool.py while doing _setup_queues()

self._inqueue = self._ctx.SimpleQueue() # creates 2 pipes
self._outqueue = self._ctx.SimpleQueue() # creates 2 pipes

and then 2 for each process in the Pool.

When joining, it closes pipes for each of the process but not 4 for _inqueue and _outqueue.
Those become open forever till the MAIN process is closed.
Code to reproduce:

import billiard as mp
import os

def f(a):
    return a + 1


if __name__ == '__main__':
    pid = os.getpid()
    get_number_of_conns = os.popen(f'ls -l /proc/{pid}/fd | wc -l').read()
    print(f'At the beginning we only have {get_number_of_conns.strip()} connections')
    for i in range(10):
        # creating a pool
        pool = mp.Pool(mp.cpu_count() - 1)
        # running a job
        result = pool.map(f, range(5))
        # closing the pool and joining
        pool.close()
        pool.join()
        # getting number of open connections
        get_number_of_conns = os.popen(f'ls -l /proc/{pid}/fd | wc -l').read()
        print(f'Open connections: {get_number_of_conns.strip()}')
Dominant language
Python
Stars
436
Forks
273
Avg merge
4d 19h
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from celery/billiard

All issues in celery/billiard

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.