Worker executes task after rejecting it when ETA timestamp conversion fails
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 82/100
Research direction
The issue identifies celery/worker/strategy.py and the ETA conversion exception path. Read that path first and trace what happens after req.reject(requeue=False). Add a regression test for an ETA conversion failure, then confirm the rejected request is not reserved or executed.
Written by the indexing model from the issue text.
Description
Checklist
- I have verified that the issue exists against the
mainbranch of Celery. - This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.
- I have read the relevant section in the
[contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)
on reporting bugs. - I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
for similar or identical bug reports. - I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
for existing proposed fixes. - I have checked the [commit log](https://github.com/celery/celery/commits/main)
to find out if the bug was already fixed in the main branch. - I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway). - I have tried to reproduce the issue with [pytest-celery](https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html) and added the reproduction script below.
Mandatory Debugging Information
- I have included the output of
celery -A proj reportin the issue.
(if you are not able to do this, then at least specify the Celery
version affected). - I have verified that the issue exists against the
mainbranch of Celery. - I have included the contents of
pip freezein the issue. - I have included all the versions of all the external dependencies required
to reproduce this bug.
Optional Debugging Information
- I have tried reproducing the issue on more than one Python version
and/or implementation. - I have tried reproducing the issue on more than one message broker and/or
result backend. - I have tried reproducing the issue on more than one version of the message
broker and/or result backend. - I have tried reproducing the issue on more than one operating system.
- I have tried reproducing the issue on more than one workers pool.
- I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled. - I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
- None found.
Possible Duplicates
- None found.
Environment & Settings
Celery version: 5.7.0b1
celery report Output:
software -> celery:5.7.0b1 (collider) kombu:5.7.0b1 py:3.13.15
billiard:4.3.1 py-amqp:5.4.1
platform -> system:Windows arch:64bit, WindowsPE
kernel version:11 imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled
deprecated_settings: None
The issue was reproduced against the current main branch corresponding to
Celery 5.7.0b1.
Steps to Reproduce
Required Dependencies
- Python: 3.13.15
- Celery: 5.7.0b1 / current
main - Kombu: 5.7.0b1
- Billiard: 4.3.1
- py-amqp: 5.4.1
- OS: Windows 11, 64-bit
- Broker: The behavior can be reproduced with an in-process
memory://broker; thecelery reportabove shows the development environment's default AMQP transport. - Result backend: Not required
Python Packages
The relevant dependency versions from celery report are:
celery==5.7.0b1
kombu==5.7.0b1
billiard==4.3.1
py-amqp==5.4.1
Python 3.13.15
A complete pip freeze can be added separately if required.
Other Dependencies
No external result backend is required.
The issue is in the worker's ETA handling in
celery/worker/strategy.py.
A memory:// broker is sufficient to reproduce the control-flow issue.
Minimally Reproducible Test Case
The problematic code path is:
try:
eta = to_timestamp(to_system_tz(req.eta))
except (OverflowError, ValueError) as exc:
error(...)
req.reject(requeue=False)
There is no return after req.reject(requeue=False).
When ETA conversion raises OverflowError or ValueError, execution therefore
continues into the normal task-processing path:
task_reserved(req)
handle(req)
For example, with the worker timezone set to Asia/Kolkata, an ETA such as:
9999-12-31T23:59:59+00:00
can raise OverflowError during timezone conversion/timestamp conversion.
The observed behavior is:
Couldn't convert ETA ...
Request.reject(requeue=False)
task_reserved(req)
handle(req)
The task is consequently executed immediately even though its message has
already been rejected.
Expected Behavior
When ETA conversion fails with OverflowError or ValueError, the worker
should reject the message and stop processing that request.
After:
req.reject(requeue=False)
the worker should not reserve or execute the task.
Actual Behavior
The worker rejects the message but then continues processing the same request.
As a result, the task can execute immediately despite having just been rejected
with requeue=False.
The problematic control flow is:
except (OverflowError, ValueError) as exc:
error(...)
req.reject(requeue=False)
# execution incorrectly continues
task_reserved(req)
handle(req)
The minimal fix is to return immediately after rejecting the request:
except (OverflowError, ValueError) as exc:
error(...)
req.reject(requeue=False)
+ return
I reproduced the issue against the current main branch. The regression tests
fail on the original implementation and pass with this control-flow fix.
- Dominant language
- Python
- Stars
- 29k
- Forks
- 5.2k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 125
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from celery/celery
-
`saferepr` drops the closing bracket when the last element is truncated (affects task `argsrepr/kwargsrepr` )Possibly taken @auvipy claimed this 13 days ago. OpenIssue Type: Bug Report
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
celery/celery#10760 · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
celery/celery#10893 · 2 comments ·
Maintainers usually reply within 1 day
-
5.7 regression: broker reconnect clears the shared worker timer, silently dropping soft/hard time limits of running tasksPossibly taken @ariachen-dev claimed this today. OpenIssue Type: Regression
Difficulty 4/5 3-5 days Newbie friendliness 35/100
celery/celery#10888 · 3 comments ·
Maintainers usually reply within 1 day
-
Request.revoked() ignores later stamped headers: revoke_by_stamped_headers fails to revoke a task when an earlier flagged header doesn't matchPossibly taken @Namraa310806 claimed this 1 day ago. OpenIssue Type: Bug Report
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
chain(..., chord).on_error() never calls the errback when the chord's header contains a nested chordPossibly taken @subham-hq claimed this 15 days ago. OpenIssue Type: Bug Report
Difficulty 4/5 3-5 days Newbie friendliness 20/100
Maintainers usually reply within 1 day
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
Status: Triaged Type: Enhancement
Difficulty 1/5 Under an hour Newbie friendliness 82/100
canonical/craft-store#448 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Maintainers usually reply within 1 day
-
good first issue research size:s skill:numerics sub-issue
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
SKaiNET-developers/SKaiNET#1372 ·
Maintainers usually reply within 1 day
-
debt
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/vscode-flake8#601 ·
Maintainers usually reply within 1 day