Bucket deletion blocked by "bucket not empty" retries only on exponential backoff, with no way to trigger a retry short of a spec change

Aperta
  • Qualcosa non va?
#145 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
65/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
backend, devops, infrastructure

Direzione di ricerca

Start with the update predicate in internal/controller/bucket/controller.go, then trace finalizer errors through internal/controller/bucket/finalizer.go and internal/helpers/controller.go. Reproduce the failed deletion and observe retry timing and update filtering. Done means recoverable bucket-deletion failures retry predictably after the bucket is emptied, with an actionable condition and consistent behavior across the named finalizers.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

What happens

With bucketDeletionEnabled: true on the S3Instance, deleting a Bucket whose S3 bucket still holds objects fails, as expected:

Bucket deletion has failed: The bucket you tried to delete is not empty

The CR keeps its deletionTimestamp and the s3.onyxia.sh/finalizer. After the user empties the bucket, the obvious expectation is that the operator notices soon and finishes the deletion. It doesn't. There are two separate problems:

  1. Retries fall back to controller-runtime's default exponential backoff, capped at 1000 s. In our case the attempts came at 16:48:45, 16:49:26, 16:50:07 and 16:59:40 UTC, and the gaps keep growing to about 16.7 minutes. The only feedback is a Reconciled=False / DeletionFailure condition. Its lastTransitionTime rightly stays unchanged, so from outside the Bucket simply looks stuck.
  2. Nothing but a spec change triggers a new attempt. The Bucket controller's event filter drops every update that doesn't change metadata.generation. Adding or changing an annotation or label does nothing, so the only way to retry now is a meaningless spec edit. That's what people end up doing, and it's not obvious.
Where (v0.12.0, 41b84ba)

The same pattern appears in the Path, Policy and S3User finalizers.

Reproduce
  1. Use an S3Instance with bucketDeletionEnabled: true.
  2. Create a Bucket and put an object in it.
  3. kubectl delete bucket <name>. The deletion fails with "not empty", as expected.
  4. Delete the object.
  5. Watch the operator logs: the next attempt comes only when the backoff expires, which after a few failures means minutes. kubectl annotate bucket <name> foo=bar doesn't trigger one; a spec change does.
Suggestions
  • Retry a known, recoverable finalizer failure on a short fixed interval. For "bucket not empty", return ctrl.Result{RequeueAfter: 30 * time.Second}, nil, after setting the condition, instead of returning the error. The retry interval then stays constant and predictable.
  • Make the condition message actionable. For example: "bucket is not empty; delete its objects (including versions) and the operator will retry".
  • Optionally, let deleting resources through the event filter. Reconciling on any update while deletionTimestamp is set would make a label or annotation change a supported manual trigger. (predicate.Or(predicate.GenerationChangedPredicate{}, predicate.AnnotationChangedPredicate{}) would do the same in general.)
  • Possibly add an opt-in on the S3Instance to empty a bucket before deleting it, for setups that already treat CR deletion as data deletion. This is a larger design question and could be a separate issue.

Environment: s3-operator v0.12.0 (inseefrlab/s3-operator:v0.12.0, controller-runtime v0.20.1), with Silo (pgsty's MinIO fork, pgsty/silo:RELEASE.2026-09-16T00-00-00Z, s3Provider: minio) behind the S3Instance, on Kubernetes 1.36. The "not empty" error is the S3 server's standard BucketNotEmpty response to DeleteBucket, so this should reproduce the same way on upstream MinIO.

Lingua principale
Go
Stelle
66
Fork
16
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Include un Dockerfile o un file Docker Compose
  • Nessun modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di InseeFrLab/s3-operator

Tutte le issue di InseeFrLab/s3-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.