5.7 regression: worker dies with AttributeError ('NoneType' object has no attribute 'hostname') when the Redis broker restarts while a prefetched task waits

Start with the traceback: on_connection_error_after_connected in celery/worker/consumer/consumer.py calls connection.close(), which reaches _connparams in kombu/transport/redis.py where conninfo.hostname is None. Run repro.py against a Redis container with the bounce loop to confirm the crash. Done when the worker survives every bounce and repro.py prints bounce N: ok for each one.

Checklist

• I have verified that the issue exists against the main branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the main branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
• I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the main branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• #10242 (fix for #9705): added the connection.close() after collect() in on_connection_error_after_connected
  that this crash goes through.
• #10888: another 5.7 broker-reconnect regression (timer entries), already fixed; different symptom.
• celery/kombu#2492, celery/kombu#2630: the Redis reconnect hang that kombu 5.7.0 fixes. With celery 5.6.3 +
  kombu 5.7.0 the same bounce test passes.

Possible Duplicates

• None

Environment & Settings

Celery version: 5.7.0 (collider); also reproduced on main at 1a48583d40 with kombu main at 38fbf78479

software -> celery:5.7.0 (collider) kombu:5.7.0 py:3.14.4
            billiard:4.3.2 redis:6.4.0
platform -> system:Linux arch:64bit
            kernel version:6.8.0-1064-azure imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://127.0.0.1:6379/1

broker_url: 'redis://:********@127.0.0.1:6379/0'
result_backend: 'redis://127.0.0.1:6379/1'
deprecated_settings: None

Steps to Reproduce

Required Dependencies

• Minimal Python Version: Unknown; reproduced on 3.14.4
• Minimal Celery Version: 5.7.0rc1 (5.6.3 is not affected)
• Minimal Kombu Version: 5.7.0rc1 (celery 5.7.0 requires kombu >= 5.7.0)
• Minimal Broker Version: Unknown; reproduced on Redis 7.2.17 (production: ElastiCache Valkey 7.2)
• Minimal Result Backend Version: Unknown; Redis 7.2.17 (same server as the broker)
• Minimal OS and/or Kernel Version: Unknown; reproduced on Linux 6.8.0
• Minimal Broker Client Version: Unknown; reproduced on redis-py 6.4.0
• Minimal Result Backend Client Version: Unknown; reproduced on redis-py 6.4.0

Python Packages

amqp==5.4.2
billiard==4.3.2
celery==5.7.0
click==8.5.1
click-didyoumean==0.3.1
click-plugins==1.1.1.2
click-repl==0.4.1
kombu==5.7.0
packaging==26.3
prompt-toolkit==3.0.53
python-dateutil==2.9.0.post0
redis==6.4.0
six==1.17.0
typing-extensions==4.16.0
tzlocal==5.4.5
vine==5.1.0
wcwidth==0.9.4

Other Dependencies

Minimally Reproducible Test Case

pip install "celery[redis]==5.7.0" "redis==6.4.0"
docker run -d --name repro-redis -p 127.0.0.1:6379:6379 redis:7.2 redis-server --appendonly yes
celery -A repro worker -c 2 -l INFO     # terminal 1
python repro.py                          # terminal 2

import subprocess
import sys
import time

import redis
from celery import Celery

URL = "redis://127.0.0.1:6379"
CONTAINER = "repro-redis"

app = Celery("repro", broker=URL + "/0", backend=URL + "/1")


@app.task
def sleep_task(secs):
    time.sleep(secs)
    return secs


@app.task
def ping_task(n):
    return n


def wait_for_redis():
    for _ in range(100):
        try:
            redis.Redis.from_url(URL).ping()
            return
        except redis.ConnectionError:
            time.sleep(0.2)


if __name__ == "__main__":
    for i in range(int(sys.argv[1]) if len(sys.argv) > 1 else 30):
        # Three tasks for two pool processes: two run, the third waits prefetched (unacked) in the worker.
        busy = [sleep_task.delay(8) for _ in range(3)]
        time.sleep(1)
        subprocess.run(["docker", "restart", "-t", "0", CONTAINER], check=True, capture_output=True)
        wait_for_redis()
        time.sleep(3)
        try:
            ping_task.delay(i).get(timeout=60)
            print(f"bounce {i}: ok", flush=True)
        except Exception as exc:
            print(f"bounce {i}: task not consumed within 60 s ({exc!r}) - check the worker log", flush=True)
            sys.exit(1)
        for r in busy:  # let the worker go idle before the next bounce
            try:
                r.get(timeout=60)
            except Exception:
                pass
    print("no failure")

Expected Behavior

The worker logs the lost connection, reconnects and keeps consuming, as celery 5.6.3 does with the same kombu 5.7.0.
repro.py prints bounce N: ok for every bounce.

Actual Behavior

The worker exits on the first bounce. That happened in 2 of 2 runs on 5.7.0 and 2 of 2 on main. The task
sent after the restart is never consumed. Under Kubernetes the pod restarts, and tasks that were running in the
pool are lost.

WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection... (ConnectionError("Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer')"))
INFO/MainProcess] Restoring 1 unacknowledged message(s)
CRITICAL/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'hostname'")
Traceback (most recent call last):
  ...
redis.exceptions.ConnectionError: Error while reading from 127.0.0.1:6379 : (104, 'Connection reset by peer')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "celery/worker/worker.py", line 271, in start
    self.blueprint.start(self)
  File "celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "celery/worker/consumer/consumer.py", line 427, in start
    self.on_connection_error_after_connected(exc)
  File "celery/worker/consumer/consumer.py", line 487, in on_connection_error_after_connected
    ignore_errors(connection, connection.close)
  File "kombu/common.py", line 334, in ignore_errors
    return fun(*args, **kwargs)
  File "kombu/connection.py", line 456, in release
    self._close()
  File "kombu/connection.py", line 422, in _close
    self._do_close_self()
  File "kombu/connection.py", line 412, in _do_close_self
    self.maybe_close_channel(self._default_channel)
  File "kombu/connection.py", line 404, in maybe_close_channel
    channel.close()
  File "kombu/transport/redis.py", line 1909, in close
    super().close()
  File "kombu/transport/virtual/base.py", line 806, in close
    self._qos.restore_unacked_once()
  File "kombu/transport/virtual/base.py", line 300, in restore_unacked_once
    unrestored = self.restore_unacked()
  File "kombu/transport/redis.py", line 494, in restore_unacked
    with self.channel.conn_or_acquire(client) as client:
  File "kombu/transport/redis.py", line 2089, in conn_or_acquire
    yield self._create_client()
  File "kombu/transport/redis.py", line 2056, in _create_client
    client_kwargs = {'connection_pool': self.pool}
  File "kombu/transport/redis.py", line 2094, in pool
    self._pool = self._get_pool()
  File "kombu/transport/redis.py", line 2065, in _get_pool
    params = self._connparams(asynchronous=asynchronous)
  File "kombu/transport/redis.py", line 1959, in _connparams
    'host': conninfo.hostname or '127.0.0.1',
AttributeError: 'NoneType' object has no attribute 'hostname'

The same thing happens in a longer harness: prefork -c 2 --max-tasks-per-child 2, 30 bounces per run, mixing
docker restart, kill + start and CLIENT KILL. Every run on 5.7.0 and 5.7.0rc1 crashed when tasks were in
flight. celery 5.6.3 + kombu 5.7.0 had no crash in 120 bounces. With an idle worker, 5.7.0 passed 90 of 90 bounces.
It only crashed on restarts, never on CLIENT KILL.

Cause, as far as I can tell. Since #10242, on_connection_error_after_connected calls
self.connection.collect() and then connection.close().

1. While the server is down, collect() → _do_close_self() → maybe_close_channel(default_channel) fails part-way.
   The Redis Channel.close() raises ConnectionError in the fanout queue_delete before it reaches
   super().close(), and maybe_close_channel swallows that. The channel stays open with its unacked messages,
   and _default_channel is not cleared.
2. collect() then runs _do_close_transport(), which sets transport.client = None.
3. connection.close() retries the default channel close. By then the server is often back, so this time the close
   reaches restore_unacked(), which needs transport.client to build a pool.
4. AttributeError is not a connection or channel error, so ignore_errors lets it through.

With CLIENT KILL the server stays up, so step 1 succeeds and the later close() is a no-op. 5.6.3 only calls
collect().

The same self.connection = None also logs a non-fatal
Error on restarting Gossip: AttributeError("'NoneType' object has no attribute 'connection_errors'") on many
bounces: ConsumerStep._close (celery/bootsteps.py:415) calls ignore_errors(c.connection, channel.close) with
c.connection already None.

Fix we tested. Guard the new close the way collect() is guarded just above it:

         connection, self.connection = self.connection, None
         if connection:
-            ignore_errors(connection, connection.close)
+            try:
+                connection.close()
+            except Exception:  # pylint: disable=broad-except
+                pass

With it there were no crashes in 90 of 90 busy bounces in the harness, or in 8 of 8 bounces of the repro. The worker
then behaves like 5.6.3, including a pre-existing kombu quirk: the waiting message is not restored at the reconnect.
It stays in the unacked hash until visibility_timeout, 1 h by default (8 entries after 8 bounces with the patch;
7 on 5.6.3 + kombu 5.7.0).

Is this guard the fix you'd want here, or would you rather fix it in kombu? There, collect() could clear
_default_channel, and the Redis Channel.close() could call super().close() in a finally, which might also
restore the waiting message straight away. Happy to open a PR with a test for either.