Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Redis task worker leaks locks when it fails to claim a task, leaving it WAITING forever

Đang mở
  • Có gì không ổn?
#8,280 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
42/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python, redis
Lĩnh vực
backend, databases

Hướng nghiên cứu

Start in pulpcore.tasking.redis_worker at the dispatch loop where Redis task and resource locks are acquired before the task is claimed. Trace the failure path and lock reconciliation, then check the related worker tests; the issue does not name specific test files. Done means a failed claim releases acquired locks, reports the failure, and leaves the task able to be picked up again.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Issue

Version

  • Redis-based task worker (pulpcore.tasking.redis_worker)
  • pulp_rpm 3.39.2
  • Deployed as a containerized service on OpenShift (multiple pulp-worker replicas, shared Redis for locks).

Describe the bug

A task can get permanently stuck in waiting because a worker acquires its Redis locks but then fails to claim the task (never sets app_lock / transitions it to running), and the failure path does not release the locks it acquired. Because the worker that leaked the locks stays alive, none of the existing cleanup paths ever reclaim them, so the task waits forever and holds its exclusive resource locks — blocking every other task queued on the same resource.

In the Redis worker's dispatch loop the lock acquisition and the task claim are not atomic:

  1. Worker pulls a task from the queue.
  2. Worker acquires the Redis task lock (task:<uuid>) and the resource locks (pulp:resource_lock:<prn>).
  3. Worker attempts to claim the task — set the DB app_lock and move it to running (which is also when it logs Starting task …).
  4. If step 3 fails, the locks from step 2 are not released.

The leaked locks have ttl -1 (no expiry) and are held by a worker that is still running. This makes them invisible to every reclaim mechanism:

  • The periodic redis_worker lock reconciliation only reclaims locks whose owner is dead. The leaking worker is alive, so its locks are skipped. (We see continuous Reconciling Redis locks for N orphan owner(s) across workers, and the count never drains these.)
  • Nothing fails over, because the task lock being present signals "a worker is handling this," so no other worker re-picks the task.
  • The claim failure in our case produced no log line and no traceback — it was swallowed — so there is no operational signal that it happened. The only trace is the leaked lock itself.

We added a diagnostic endpoint that classifies these as abandoned (held by a live worker for a task that isn't running), as distinct from orphaned (held by a dead worker). A real example:

abandoned_task_locks:
  lock_key:   task:<task-uuid>
  holder:     1@pulp-worker-...   (alive, heartbeating)
  ttl:        -1
  task_state: waiting
  reason:     "Task is waiting with no app_lock; worker acquired the Redis lock but did not claim the task"

abandoned_resource_locks:
  lock_key:         pulp:resource_lock:prn:rpm.rpmrepository:<repo-uuid>
  holder:           1@pulp-worker-...   (alive)
  ttl:              -1
  abandoned_by_task: <task-uuid>
  reason:           "Resource lock held by worker that abandoned task <task-uuid>"

The affected task (pulp_rpm.app.tasks.signing.signed_add_and_remove) had unblocked_at: null, started_at: null, app_lock.locked: false, yet both its task lock and the repository's exclusive resource lock were held by a live worker. It sat in waiting for ~33 hours, and a later publish task on the same repository was in turn stuck behind the held exclusive lock.

To Reproduce

Exact triggering of the claim failure is racy, but the failure mode is reproducible by interrupting a worker between acquiring the Redis locks and claiming the task:

  1. Dispatch a task that reserves an exclusive resource (e.g. an RPM repo modify/).
  2. In the dispatch loop, let the worker acquire the task + resource locks.
  3. Cause the claim step (set app_lock / move to running) to raise/abort — e.g. a transient DB error — while the worker process keeps running.
  4. Observe: task stays waiting, app_lock unset, but task:<uuid> and pulp:resource_lock:<prn> remain held by the still-alive worker with ttl -1.
  5. No reconciliation reclaims the locks (holder is alive); the task never progresses; subsequent tasks on the resource block.

Expected behavior

Acquiring the locks and claiming the task should be atomic, or the failure path must release any locks acquired in step 2. Specifically:

  • If the claim step fails, the worker should release the task lock and all resource locks it just acquired (ideally via a try/finally / context manager around acquire→claim).
  • The claim failure should be logged (with traceback), not swallowed.
  • Lock reconciliation should also handle the abandoned case — a lock held by a live worker for a task that is waiting with no app_lock is safe to release and re-queue, not just locks held by dead workers.
  • Consider a TTL / lease on Redis locks as a backstop so a leaked lock cannot persist indefinitely.

Additional context

Current operational workaround is to force-release the task's Redis locks, after which a worker re-picks the task and it (and anything blocked behind it) drains. This is a manual, out-of-band fix; the underlying leak recurs.

Ngôn ngữ chính
Python
Star
619
Fork
176
Merge trung bình
17 giờ 14 phút
Pull request đã merge (30 ngày)
127

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của pulp/pulpcore

Tất cả issue của pulp/pulpcore

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.