billiard.Manager broken in billiard 3.5.0.3 in python 2.7, used to work in 3.3.0.21

Open
  • Is something wrong?
#242 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
backend, operating-systems

Research direction

Reproduce the three-statement Manager and Queue example under Python 2.7.5, then inspect billiard/process.py, billiard/connection.py, and billiard/managers.py at the reported _tempdir, challenge, and BaseProxy._mutex paths. Done means Manager() and manager.Queue() work without the reported AttributeError, HMAC failure, or EOFError.

Written by the indexing model from the issue text.

Description

Here's a trivial example of using a Manager that works with the original multiprocessing, and used to work in billiard 3.3.0.21, but no longer works with 3.5.0.3 running in linux on Python 2.7.5:

import billiard
m = billiard.Manager()
q = m.Queue()

Assume I'm not calling the above inside a celery task, I'm executing the above three statements in a new python process.

with billiard 3.5.0.3 running in linux on Python 2.7.5 , we immediately see a crash:

>>> import billiard
>>> m = billiard.Manager()
Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 529, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 158, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 483, in __init__
    address = address or arbitrary_address(family)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 93, in arbitrary_address
    return tempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
  File "multiprocessing/util.py", line 141, in get_temp_dir
    current_process()._tempdir = tempdir
AttributeError: can't set attribute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/billiard/context.py", line 94, in Manager
    m.start()
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 507, in start
    self._address = reader.recv()
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 280, in recv
    buf = self._recv_bytes()
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 458, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 432, in _recv
    raise EOFError
EOFError

I tried to debug this. So far I can see the following issues. Each subsequent issue is revealed by patching to fix or work around the previous issues:

  1. billiard.process.BaseProcess defines a read-only _tempdir property, but billiard re-uses multiprocessing.util.get_temp_dir() which assumes a processes' _tempdir is writeable. This is simple to fix.

  2. billiard.connection's deliver_challenge and answer_challenge functions appear to break under python 2.7. The calls to hmac.new(authkey, message, 'md5').digest() fail in python 2.7 as passing a value of str type for the third argument is not supported. importing hashlib and passing hashlib.md5 resolves this in python 2.7 and reads as if it should work in python 3. Ref: https://docs.python.org/2/library/hmac.html#hmac.new ; https://docs.python.org/3/library/hmac.html#hmac.new .

  3. calling manager.Queue() triggers a crash as billiard.managers tries to do with BaseProxy._mutex:, which fails, as the _mutex attribute is a multiprocessing.util.ForkAwareThreadLock object which doesn't implement the context manager protocol. See: https://github.com/celery/billiard/blame/78a5b4592446466afe1020b49b01918cdeaeb9f0/billiard/managers.py#L693 .

I stopped investigating after hitting the third issue, so there may be other regressions as well.

I'm not hitting these billiard issues directly with celery -- I have some legacy application code that was originally written to use multiprocessing Managers and Queues, then previously we replaced all of our dependencies on multiprocessing with dependencies on billiard to work around issues with celery tasks launching subprocesses after migrating the application from windows to linux (the old "Tasks are not allowed to start subprocesses" celery issue: https://github.com/celery/celery/issues/1709 ), but now these recent billiard changes prevent us from upgrading to newer versions of billiard and celery until we rewrite our application code to stop depending upon parts of the multiprocessing API that billiard breaks support for.

Dominant language
Python
Stars
436
Forks
273
Avg merge
4d 19h
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from celery/billiard

All issues in celery/billiard

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.