Bucket deletion blocked by "bucket not empty" retries only on exponential backoff, with no way to trigger a retry short of a spec change

Đang mở
  • Có gì không ổn?
#145 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
65/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go, kubernetes
Lĩnh vực
backend, devops, infrastructure

Hướng nghiên cứu

Start with the update predicate in internal/controller/bucket/controller.go, then trace finalizer errors through internal/controller/bucket/finalizer.go and internal/helpers/controller.go. Reproduce the failed deletion and observe retry timing and update filtering. Done means recoverable bucket-deletion failures retry predictably after the bucket is emptied, with an actionable condition and consistent behavior across the named finalizers.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

What happens

With bucketDeletionEnabled: true on the S3Instance, deleting a Bucket whose S3 bucket still holds objects fails, as expected:

Bucket deletion has failed: The bucket you tried to delete is not empty

The CR keeps its deletionTimestamp and the s3.onyxia.sh/finalizer. After the user empties the bucket, the obvious expectation is that the operator notices soon and finishes the deletion. It doesn't. There are two separate problems:

  1. Retries fall back to controller-runtime's default exponential backoff, capped at 1000 s. In our case the attempts came at 16:48:45, 16:49:26, 16:50:07 and 16:59:40 UTC, and the gaps keep growing to about 16.7 minutes. The only feedback is a Reconciled=False / DeletionFailure condition. Its lastTransitionTime rightly stays unchanged, so from outside the Bucket simply looks stuck.
  2. Nothing but a spec change triggers a new attempt. The Bucket controller's event filter drops every update that doesn't change metadata.generation. Adding or changing an annotation or label does nothing, so the only way to retry now is a meaningless spec edit. That's what people end up doing, and it's not obvious.
Where (v0.12.0, 41b84ba)

The same pattern appears in the Path, Policy and S3User finalizers.

Reproduce
  1. Use an S3Instance with bucketDeletionEnabled: true.
  2. Create a Bucket and put an object in it.
  3. kubectl delete bucket <name>. The deletion fails with "not empty", as expected.
  4. Delete the object.
  5. Watch the operator logs: the next attempt comes only when the backoff expires, which after a few failures means minutes. kubectl annotate bucket <name> foo=bar doesn't trigger one; a spec change does.
Suggestions
  • Retry a known, recoverable finalizer failure on a short fixed interval. For "bucket not empty", return ctrl.Result{RequeueAfter: 30 * time.Second}, nil, after setting the condition, instead of returning the error. The retry interval then stays constant and predictable.
  • Make the condition message actionable. For example: "bucket is not empty; delete its objects (including versions) and the operator will retry".
  • Optionally, let deleting resources through the event filter. Reconciling on any update while deletionTimestamp is set would make a label or annotation change a supported manual trigger. (predicate.Or(predicate.GenerationChangedPredicate{}, predicate.AnnotationChangedPredicate{}) would do the same in general.)
  • Possibly add an opt-in on the S3Instance to empty a bucket before deleting it, for setups that already treat CR deletion as data deletion. This is a larger design question and could be a separate issue.

Environment: s3-operator v0.12.0 (inseefrlab/s3-operator:v0.12.0, controller-runtime v0.20.1), with Silo (pgsty's MinIO fork, pgsty/silo:RELEASE.2026-09-16T00-00-00Z, s3Provider: minio) behind the S3Instance, on Kubernetes 1.36. The "not empty" error is the S3 server's standard BucketNotEmpty response to DeleteBucket, so this should reproduce the same way on upstream MinIO.

Ngôn ngữ chính
Go
Star
66
Fork
16
Không có pull request nào được merge trong 30 ngày

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Có Dockerfile hoặc tệp Docker Compose
  • Không có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của InseeFrLab/s3-operator

Tất cả issue của InseeFrLab/s3-operator

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.