Billiard is leaving pipes open

Open
  • Is something wrong?
#329 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
operating-systems

Research direction

The report names no source file or test. Start by reproducing the long-running pipeline with Billiard's multiprocessing imports, Celery, and Luigi workers, then inspect open pipe counts over time. Done means the count does not continually increase and the run avoids OSError: [Errno 24] Too many open files.

Written by the indexing model from the issue text.

Description

I am using Celery to initiate long running pipelines and I override multiprocessing imports with Biliard. I notice as a pipeline runs, the number of open files (pipes) steadly increases and never decreases. This occurs until tens of thousands of pipes are opened and I get the error OSError: [Errno 24] Too many open files.
I ran this pipeline outside of Celery without overriding multiprocessing imports and this issue does not occur. Processes are spawned by Luigi workers and I am using the latest version of Celery.

Dominant language
Python
Stars
436
Forks
273
Avg merge
4d 19h
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from celery/billiard

All issues in celery/billiard

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.