Bucket deletion blocked by "bucket not empty" retries only on exponential backoff, with no way to trigger a retry short of a spec change

Abierto
  • ¿Algo no está bien?
#145 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
65/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, kubernetes
Área
backend, devops, infrastructure

Línea de trabajo

Start with the update predicate in internal/controller/bucket/controller.go, then trace finalizer errors through internal/controller/bucket/finalizer.go and internal/helpers/controller.go. Reproduce the failed deletion and observe retry timing and update filtering. Done means recoverable bucket-deletion failures retry predictably after the bucket is emptied, with an actionable condition and consistent behavior across the named finalizers.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happens

With bucketDeletionEnabled: true on the S3Instance, deleting a Bucket whose S3 bucket still holds objects fails, as expected:

Bucket deletion has failed: The bucket you tried to delete is not empty

The CR keeps its deletionTimestamp and the s3.onyxia.sh/finalizer. After the user empties the bucket, the obvious expectation is that the operator notices soon and finishes the deletion. It doesn't. There are two separate problems:

  1. Retries fall back to controller-runtime's default exponential backoff, capped at 1000 s. In our case the attempts came at 16:48:45, 16:49:26, 16:50:07 and 16:59:40 UTC, and the gaps keep growing to about 16.7 minutes. The only feedback is a Reconciled=False / DeletionFailure condition. Its lastTransitionTime rightly stays unchanged, so from outside the Bucket simply looks stuck.
  2. Nothing but a spec change triggers a new attempt. The Bucket controller's event filter drops every update that doesn't change metadata.generation. Adding or changing an annotation or label does nothing, so the only way to retry now is a meaningless spec edit. That's what people end up doing, and it's not obvious.
Where (v0.12.0, 41b84ba)

The same pattern appears in the Path, Policy and S3User finalizers.

Reproduce
  1. Use an S3Instance with bucketDeletionEnabled: true.
  2. Create a Bucket and put an object in it.
  3. kubectl delete bucket <name>. The deletion fails with "not empty", as expected.
  4. Delete the object.
  5. Watch the operator logs: the next attempt comes only when the backoff expires, which after a few failures means minutes. kubectl annotate bucket <name> foo=bar doesn't trigger one; a spec change does.
Suggestions
  • Retry a known, recoverable finalizer failure on a short fixed interval. For "bucket not empty", return ctrl.Result{RequeueAfter: 30 * time.Second}, nil, after setting the condition, instead of returning the error. The retry interval then stays constant and predictable.
  • Make the condition message actionable. For example: "bucket is not empty; delete its objects (including versions) and the operator will retry".
  • Optionally, let deleting resources through the event filter. Reconciling on any update while deletionTimestamp is set would make a label or annotation change a supported manual trigger. (predicate.Or(predicate.GenerationChangedPredicate{}, predicate.AnnotationChangedPredicate{}) would do the same in general.)
  • Possibly add an opt-in on the S3Instance to empty a bucket before deleting it, for setups that already treat CR deletion as data deletion. This is a larger design question and could be a separate issue.

Environment: s3-operator v0.12.0 (inseefrlab/s3-operator:v0.12.0, controller-runtime v0.20.1), with Silo (pgsty's MinIO fork, pgsty/silo:RELEASE.2026-09-16T00-00-00Z, s3Provider: minio) behind the S3Instance, on Kubernetes 1.36. The "not empty" error is the S3 server's standard BucketNotEmpty response to DeleteBucket, so this should reproduce the same way on upstream MinIO.

Lenguaje dominante
Go
Estrellas
66
Forks
16
Sin PR fusionados en 30 d

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Incluye un Dockerfile o un archivo de Docker Compose
  • Sin plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de InseeFrLab/s3-operator

Todos los issues de InseeFrLab/s3-operator

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.