Redis task worker leaks locks when it fails to claim a task, leaving it WAITING forever

Start in pulpcore.tasking.redis_worker at the dispatch loop where Redis task and resource locks are acquired before the task is claimed. Trace the failure path and lock reconciliation, then check the related worker tests; the issue does not name specific test files. Done means a failed claim releases acquired locks, reports the failure, and leaves the task able to be picked up again.

Version

• Redis-based task worker (pulpcore.tasking.redis_worker)
• pulp_rpm 3.39.2
• Deployed as a containerized service on OpenShift (multiple pulp-worker replicas, shared Redis for locks).

Describe the bug

A task can get permanently stuck in waiting because a worker acquires its Redis locks but then fails to claim the task (never sets app_lock / transitions it to running), and the failure path does not release the locks it acquired. Because the worker that leaked the locks stays alive, none of the existing cleanup paths ever reclaim them, so the task waits forever and holds its exclusive resource locks — blocking every other task queued on the same resource.

In the Redis worker's dispatch loop the lock acquisition and the task claim are not atomic:

1. Worker pulls a task from the queue.
2. Worker acquires the Redis task lock (task:<uuid>) and the resource locks (pulp:resource_lock:<prn>).
3. Worker attempts to claim the task — set the DB app_lock and move it to running (which is also when it logs Starting task …).
4. If step 3 fails, the locks from step 2 are not released.

The leaked locks have ttl -1 (no expiry) and are held by a worker that is still running. This makes them invisible to every reclaim mechanism:

• The periodic redis_worker lock reconciliation only reclaims locks whose owner is dead. The leaking worker is alive, so its locks are skipped. (We see continuous Reconciling Redis locks for N orphan owner(s) across workers, and the count never drains these.)
• Nothing fails over, because the task lock being present signals "a worker is handling this," so no other worker re-picks the task.
• The claim failure in our case produced no log line and no traceback — it was swallowed — so there is no operational signal that it happened. The only trace is the leaked lock itself.

We added a diagnostic endpoint that classifies these as abandoned (held by a live worker for a task that isn't running), as distinct from orphaned (held by a dead worker). A real example:

abandoned_task_locks:
  lock_key:   task:<task-uuid>
  holder:     1@pulp-worker-...   (alive, heartbeating)
  ttl:        -1
  task_state: waiting
  reason:     "Task is waiting with no app_lock; worker acquired the Redis lock but did not claim the task"

abandoned_resource_locks:
  lock_key:         pulp:resource_lock:prn:rpm.rpmrepository:<repo-uuid>
  holder:           1@pulp-worker-...   (alive)
  ttl:              -1
  abandoned_by_task: <task-uuid>
  reason:           "Resource lock held by worker that abandoned task <task-uuid>"

The affected task (pulp_rpm.app.tasks.signing.signed_add_and_remove) had unblocked_at: null, started_at: null, app_lock.locked: false, yet both its task lock and the repository's exclusive resource lock were held by a live worker. It sat in waiting for ~33 hours, and a later publish task on the same repository was in turn stuck behind the held exclusive lock.

To Reproduce

Exact triggering of the claim failure is racy, but the failure mode is reproducible by interrupting a worker between acquiring the Redis locks and claiming the task:

1. Dispatch a task that reserves an exclusive resource (e.g. an RPM repo modify/).
2. In the dispatch loop, let the worker acquire the task + resource locks.
3. Cause the claim step (set app_lock / move to running) to raise/abort — e.g. a transient DB error — while the worker process keeps running.
4. Observe: task stays waiting, app_lock unset, but task:<uuid> and pulp:resource_lock:<prn> remain held by the still-alive worker with ttl -1.
5. No reconciliation reclaims the locks (holder is alive); the task never progresses; subsequent tasks on the resource block.

Expected behavior

Acquiring the locks and claiming the task should be atomic, or the failure path must release any locks acquired in step 2. Specifically:

• If the claim step fails, the worker should release the task lock and all resource locks it just acquired (ideally via a try/finally / context manager around acquire→claim).
• The claim failure should be logged (with traceback), not swallowed.
• Lock reconciliation should also handle the abandoned case — a lock held by a live worker for a task that is waiting with no app_lock is safe to release and re-queue, not just locks held by dead workers.
• Consider a TTL / lease on Redis locks as a backstop so a leaked lock cannot persist indefinitely.

Additional context

Current operational workaround is to force-release the task's Redis locks, after which a worker re-picks the task and it (and anything blocked behind it) drains. This is a manual, out-of-band fix; the underlying leak recurs.