Task randomly get stuck (between parent and child)

Open
  • Is something wrong?
#389 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
python
Domain
distributed-systems

Research direction

Start with the pool handoff path in billiard/pool.py, then trace queues.py and connection.py from the reported recv_bytes stack trace. Try to reproduce the rare parent-child stall with Celery 5.2.3 and billiard 3.6.1.0, and inspect whether a regression test can demonstrate the failure. Done means identifying and correcting the stuck handoff, with the task resubmitted or timed out rather than remaining stuck forever.

Written by the indexing model from the issue text.

Description

We use celery 5.2.3 and billiard 3.6.1.0. What we observe is the main worker receives a task (we can see task_received signal being fired) and then there is no trace of the task in the logs after that (not always and very rarely).
The main worker thinks it has sent the task on the pipe whereas the child is still stuck on read. As ack is not received by the main worker the timers (soft and hard) also don't fire. This task is now stuck forever. When we kill the child process the task is sent to another worker. The main worker meanwhile was directing new tasks to other children successfully.

This happens very randomly (after weeks of running normally) in production.

Stacktrace of the child when it was sent SIGUSR1:

Pool process <celery.concurrency.asynpool.Worker object at 0x7f1d1c1cf850> error: SoftTimeLimitExceeded()
Traceback (most recent call last):
  File ".../python/lib/python3.9/site-packages/billiard/pool.py", line 292, in __call__
    sys.exit(self.workloop(pid=pid))
  File ".../python/lib/python3.9/site-packages/billiard/pool.py", line 351, in workloop
    req = wait_for_job()
  File ".../python/lib/python3.9/site-packages/billiard/pool.py", line 473, in receive
    ready, req = _receive(1.0)
  File ".../python/lib/python3.9/site-packages/billiard/pool.py", line 445, in _recv
    return True, loads(get_payload())
  File ".../python/lib/python3.9/site-packages/billiard/queues.py", line 355, in get_payload
    return self._reader.recv_bytes()
  File ".../python/lib/python3.9/site-packages/billiard/connection.py", line 243, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File ".../python/lib/python3.9/site-packages/billiard/connection.py", line 460, in _recv_bytes
    return self._recv(size)
  File ".../python/lib/python3.9/site-packages/billiard/connection.py", line 422, in _recv
    chunk = read(handle, remaining)
  File ".../python/lib/python3.9/site-packages/billiard/pool.py", line 229, in soft_timeout_sighandler
    raise SoftTimeLimitExceeded()
billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()

The fact that when we killed this child, the task gets resubmitted to another child makes me think that the parent thought it sent the task on pipe and was waiting for ack whereas the receiver was stuck on read (forever). Could it be because of some race in the receiver or sender?

Dominant language
Python
Stars
436
Forks
273
Avg merge
4d 19h
Merged PRs (30d)
5

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from celery/billiard

All issues in celery/billiard

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.