Airflow is down: its Redis pod has been unable to start for 19 days (ImagePullBackOff)

Labels: bug airflow kubernetes priority DevOps help wanted

TL;DR

The Airflow Redis pod (airflow-redis-master-0) has been stuck and unable to start since 2026-06-13. Redis is the "post office" that Airflow uses to hand jobs to its workers, so with it down no scheduled Airflow job can actually run. The pod can't start because the container image it's told to download no longer exists at the address configured (the vendor, Bitnami, removed the old image from its public registry).

This is a separate problem from #315, but the two are linked — see Relationship to #315 below.

---

Background (plain-language)

What is Airflow? Airflow is our job scheduler. It runs recurring data tasks on a timer — reporting exports, earnings exports, tree-to-cluster assignment, map cache pre-warming, and more.

What is Redis, and why does Airflow need it? Airflow runs its jobs using a setup called the CeleryExecutor. In that setup there are two roles:

• a scheduler that decides what should run and when, and
• workers that actually do the work.

The scheduler and the workers don't talk directly. They pass jobs through a shared queue — like a post office. Redis is that post office. The scheduler drops a job into Redis; a worker picks it up from Redis and runs it. Airflow also uses Redis to keep track of each job's status.

So if Redis is down… the scheduler has nowhere to drop jobs, and workers have nothing to pick up. Jobs can't be handed off, so nothing runs.

---

Root cause

The Redis pod is trying to download a container image that no longer exists:

Image:  public.ecr.aws/bitnami/redis:5.0.14-debian-10-r32
Status: ImagePullBackOff / ErrImagePull
Error:  failed to pull and unpack image
        "public.ecr.aws/bitnami/redis:5.0.14-debian-10-r32":
        ... not found

Bitnami (the vendor that publishes this Redis image) removed older images from its free public registry (public.ecr.aws/bitnami) in 2025. The specific tag we ask for — 5.0.14-debian-10-r32 — is one of the removed ones, so the download fails with "not found."

Why did this start on June 13, 2026? The Redis pod ran fine before because its image was already cached on the old node. On June 13 the cluster nodes were rebuilt (all nodes and Airflow pods show an age of ~19 days). Redis was rescheduled onto a fresh node that had no cached copy of the image, tried to download it, and hit the "not found" error. It has been retrying and failing ever since.

Where the image comes from in our config: We do not pin the Redis image anywhere in our own repo. Airflow is installed as a Helm chart via Ansible:

• File: airflow/roles/airflow/tasks/main.yml
• Chart: airflow-stable/airflow, chart_version: 8.5.2

Our redis: section in that file only sets pod placement (affinity) — it does not choose a Redis image:

redis:
  master:
    affinity: *affinity

So the broken image tag is inherited from chart 8.5.2's built-in defaults, and those defaults point at the now-deleted Bitnami image.

There is only one Redis pod (a single StatefulSet replica) — there is no standby to take over.

---

Impact

Severity: high. Airflow has effectively not run any jobs for ~19 days.

1. No Airflow task can run. When the scheduler tries to queue a job, the hand-off to Redis times out. In the logs this shows up as:

   celery_executor Task Timeout Error [Try 1 of 3] ... [Try 3 of 3]
   Executor reports ... exited with status failed
   Executor reports task instance ... finished (failed) although the task
     says its queued. ... Was the task killed externally?
   

   The job is marked failed without its actual work ever running. We found zero successful task runs in the logs.

2. Every scheduled data pipeline is stalled, including:

   • pre_request_map_clusters — map cache pre-warming
   • assign_tree_to_cluster — keeps the clusters table up to date (the fast global-map tiles depend on this; that data is now ~19 days stale)
   • reporting / earnings / capture exports, leaderboards, species-id, etc.

   This is a silent failure — there is no obvious user-facing error, but derived data and exports quietly stop updating and drift further out of date the longer it lasts.

3. Restart churn on Airflow components. Because the pre_request_map_clusters job is configured to retry up to 15 times with a 1-second delay, each dead job re-queues ~15 times in a few seconds. Consistent with this, we observe:

   • airflow-worker-0: 553 restarts
   • airflow-scheduler: 57 restarts (killed by its own health check: UNHEALTHY - 0 alive SchedulerJob)
   • airflow-flower: OOM-killed once

Not affected: Airflow's database side is healthy — airflow-web, airflow-db-migrations, and airflow-pgbouncer are all Running with 0 restarts. The outage is isolated to the Celery message broker (Redis).

---

Relationship to #315

Issue #315 describes the tile server and Airflow scheduler being hammered by the pre_request_map_clusters job firing every 5 minutes and triggering slow database queries.

This Redis outage is currently masking that problem. Because no Airflow job can run, pre_request_map_clusters never makes its HTTP calls to the tile server. That's why, right now, pgpool looks healthy and the scheduler restart rate is far lower than #315 reported.

⚠️ Important sequencing consequence: the moment Redis is fixed, pre_request_map_clusters will start firing again every 5 minutes and the #315 symptoms will likely return. The #315 tile-server fixes (query statement_timeout and reducing the DAG retries) should be deployed before or together with the Redis fix, so we don't uncork the pre-warm job onto an unprotected tile server.

---

Possible solutions

Any of the following would get Redis running again. Listed roughly easiest-to-most-robust.

Option A — Point Redis at a working image (quickest). Override the Redis image in airflow/roles/airflow/tasks/main.yml under release_values.redis to an image that still exists. Bitnami moved its legacy images to a bitnamilegacy namespace, so a like-for-like replacement is usually available there. Something like:

redis:
  image:
    registry: docker.io
    repository: bitnamilegacy/redis
    tag: <a-valid-tag>
  master:
    affinity: *affinity

(The exact registry/repository/tag keys depend on chart 8.5.2's schema — confirm against the chart's values.yaml before applying.)

Option B — Mirror the image into our own registry (most robust against future deletions). Copy a known-good Redis image into a registry we control (e.g. DigitalOcean Container Registry) and point the chart at that. This prevents a repeat if a vendor deletes images again.

Option C — Upgrade the Airflow Helm chart. Bump chart_version from 8.5.2 to a newer release whose default Redis image still exists. This is more involved (a chart upgrade can change other settings and needs testing), but keeps us on maintained defaults.

Before starting — confirm how production is actually deployed. The file above is the source of truth only if production is reconciled from this Ansible playbook. If someone may have installed or changed the chart by hand, production could have drifted. Step 0 for whoever picks this up: confirm the live release matches this playbook (e.g. helm list -n airflow and helm get values), then decide where the fix should be applied.

---

Evidence

Finding	Source
airflow-redis-master-0 in ImagePullBackOff, Pending since 2026-06-13T07:15:42Z	kubectl get pods -n airflow / describe
Pull fails: public.ecr.aws/bitnami/redis:5.0.14-debian-10-r32 → not found	pod events
Single Redis replica, no standby	kubectl get pods -n airflow (only airflow-redis-master-0)
Scheduler/worker/flower all configured to use this Redis (REDIS_PASSWORD from secret airflow-redis)	pod describe (env)
Jobs fail at the queue hand-off: Task Timeout Error, "finished (failed) although the task says its queued… killed externally"	scheduler logs
No successful task runs found	worker/scheduler logs
airflow-worker-0: 553 restarts; airflow-scheduler: 57 restarts; airflow-flower: OOMKilled once	kubectl get pods -n airflow
Airflow DB side healthy (web, db-migrations, pgbouncer = 0 restarts)	kubectl get pods -n airflow
Redis image not pinned by us; inherited from chart airflow-stable/airflow v8.5.2 defaults	airflow/roles/airflow/tasks/main.yml