Worker stops consuming after a remote control reply fails to encode, if pool processes get replaced

Start with the traceback: celery/worker/pidbox.py on_message catches the EncodeError and resets the pidbox, and kombu/pidbox.py reply/_publish_reply is where the publish fails. The reproduction script is in the issue body and needs a Valkey or Redis broker plus --max-tasks-per-child. Done means the worker keeps consuming at its normal rate after a failing reply, shown by the per-10-second task count. The root cause is not yet identified, so expect to debug the pool and pidbox interaction first.

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• #3320 (stats() from Flower failing with EncodeError, 2018 — same kind of trigger)
• #11056 (inspect stats fails when broker transport options hold a non-JSON value — the trigger in our case)
• #10363 (changed Pidbox._close_channel() for reset cycles; part of 5.7.0 — we haven't checked whether it's
  involved, but 5.6.3 broke differently on the same trigger, see below)

Possible Duplicates

• None

Environment & Settings

Celery version: 5.7.0 (collider); also main at 1a48583d

software -> celery:5.7.0 (collider) kombu:5.7.0 py:3.13.18
            billiard:4.3.2 redis:8.1.0
platform -> system:Linux arch:64bit, ELF
            kernel version:7.0.0-1014-aws imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

broker_url: 'redis://:********@127.0.0.1:6399/0'
deprecated_settings: None
worker_prefetch_multiplier: 1
task_default_queue: 'plain'
task_ignore_result: True
broker_connection_retry_on_startup: True

Steps to Reproduce

Required Dependencies

• Minimal Python Version: Unknown (seen on 3.13.18 and 3.14.10)
• Minimal Celery Version: Unknown (seen on 5.7.0 and main 1a48583d)
• Minimal Kombu Version: Unknown (seen on 5.7.0 and main 38fbf784)
• Minimal Broker Version: Unknown (Valkey 8.1)
• Minimal Result Backend Version: N/A (results disabled)
• Minimal OS and/or Kernel Version: Unknown (Ubuntu 24.04, kernel 7.0.0-1014-aws)
• Minimal Broker Client Version: Unknown (redis-py 8.1.0)
• Minimal Result Backend Client Version: N/A

Python Packages

amqp==5.4.2
billiard @ git+https://github.com/celery/billiard@895817a456af1a56c28675dfe4e22b87b991a72a
celery @ git+https://github.com/celery/celery@1a48583d40d31a7525474f83cf543b4194e113b5
click==8.5.1
click-didyoumean==0.3.1
click-plugins==1.1.1.2
click-repl==0.4.1
kombu @ git+https://github.com/celery/kombu@38fbf784791faa46c13c92a89c45a0c5c7bcd601
packaging==26.3
prompt-toolkit==3.0.53
python-dateutil==4.0.0
redis==8.1.0
typing-extensions==4.16.0
tzdata==2026.6
tzlocal==5.4.5
vine==5.1.0
wcwidth==0.9.4

Other Dependencies

Minimally Reproducible Test Case

# app.py
import os
import time

from celery import Celery
from celery.worker.control import inspect_command

app = Celery('repro', broker='redis://127.0.0.1:6399/0')
app.conf.update(worker_prefetch_multiplier=1, task_default_queue='repro', task_ignore_result=True)


@inspect_command()
def boom(state):
    # The reply can't be encoded, so publishing it raises EncodeError outside the handler:
    # celery/worker/pidbox.py logs "Control command error" and resets the pidbox.
    # (An exception raised inside the handler would just be returned as {'error': ...}.)
    return {'value': object()}


@app.task
def work(i):
    time.sleep(0.25)
    with open('/tmp/repro.log', 'a') as f:
        f.write(f'{os.getpid()} {time.time():.3f} {i}\n')

Expected Behavior

A control command reply that can't be encoded fails only that reply; the worker keeps consuming at the same rate.

Actual Behavior

The worker logs Control command error: EncodeError(...) (traceback below) and, if pool processes are being replaced
(--max-tasks-per-child), it almost stops taking messages from the broker: Task ... received lines drop to 0–2 per
10 s while the queue still holds thousands of messages. Nothing else is logged (no WorkerLostError).

Tasks finished per 10 s (* = after the first boom):

Without process replacement the same failing replies have no visible effect, and process replacement alone (no
failing reply) is fine too.

Worker state ~110 s after the first boom (--max-tasks-per-child 10), two runs:

• main: inspect stats → prefetch_count 8, 8 pool processes, pool.writes.inqueues {'total': 8, 'active': 0};
  inspect active 0, inspect reserved 1; Redis LLEN repro 3749, HLEN unacked 1.
• 5.7.0: prefetch_count 8, 8 pool processes, pool.writes.inqueues {'total': 7, 'active': 0};
  inspect active 5, inspect reserved 2; Redis LLEN repro 3593, HLEN unacked 2.

What we ruled out while looking for the cause: the Hub.remove() calls the redis transport makes on the pidbox
disconnect only hit sockets that are already closed, and the transport never registers on_readable on a pool pipe.

With celery 5.6.3 / kombu 5.6.1 / redis-py 5.2.1 (sentinel transport, a failing inspect stats reply as the trigger)
the worker stopped consuming completely after the first failing reply, even without process replacement.

How we found it: in production (Redis via Sentinel, prefork, --max-tasks-per-child 1000) Flower sent inspect stats
on startup, the reply failed on every worker (#11056), and right after that several prefork processes of each worker
stopped getting tasks; queue latency went from ~0 to 100+ s until the workers were restarted.

[2026-11-09 15:00:20,518: ERROR/MainProcess] Control command error: EncodeError(TypeError('Object of type object is not JSON serializable'))
Traceback (most recent call last):
  File ".../kombu/serialization.py", line 41, in _reraise_errors
    yield
  File ".../kombu/serialization.py", line 220, in dumps
    payload = encoder(data)
  File ".../kombu/utils/json.py", line 63, in dumps
    return _dumps(s, cls=cls, **dict(default_kwargs, **kwargs))
  ...
TypeError: Object of type object is not JSON serializable
when serializing dict item 'value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../celery/worker/pidbox.py", line 44, in on_message
    self.node.handle_message(body, message)
  File ".../kombu/pidbox.py", line 152, in handle_message
    return self.dispatch(**body)
  File ".../kombu/pidbox.py", line 119, in dispatch
    self.reply({self.hostname: reply},
  File ".../kombu/pidbox.py", line 156, in reply
    self.mailbox._publish_reply(data, exchange, routing_key, ticket,
  File ".../kombu/pidbox.py", line 296, in _publish_reply
    producer.publish(
  ...
kombu.exceptions.EncodeError: Object of type object is not JSON serializable