StatefulSet stays deleted after a rejected create in the Recreate fallback: no retry until the operator restarts

Aperta
  • Qualcosa non va?
#2,126 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, kubernetes
Ambito
infrastructure

Direzione di ricerca

Start in statefulset-reconciler.go at shouldAbortOrContinueCreateStatefulSet (around line 442) and doCreateStatefulSet (around line 469), then trace how the Update -> Recreate path at updateStatefulSet (around line 337) propagates the create error up through worker-reconciler-chi.go:989 and worker.go:482. Reproduce with the linked kind script, focusing on why the failed create is treated as terminal instead of triggering a later reconcile/retry. Done looks like the missing StatefulSet being recreated after the webhook recovers, without an operator restart, and a regression test or repeatable repro covering the Recreate fallback.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

When the Update -> Recreate fallback deletes a StatefulSet and the following create is rejected (here by an admission webhook that fails), the StatefulSet stays deleted. The rejection is temporary, but the operator does not retry after it ends. Only an operator restart brought the StatefulSet back in our test.

#1990, #1995 and #2086 fixed the reporting of this failure: the CHI now ends as Aborted instead of Completed. The host is still lost, and nothing recovers it.

Versions
  • clickhouse-operator 0.27.4 (master has the same shouldAbortOrContinueCreateStatefulSet code)
  • kind v0.31.0 (Kubernetes v1.35.0), ClickHouse 25.8
Steps to reproduce

Full repro in a throwaway kind cluster (about 6 minutes): https://gist.github.com/valner/54b06148a436bb7489d711eee916f39f. Run bash repro.sh. The script does the following:

  1. Deploys a CHI with 1 shard and 2 replicas and waits for Completed.
  2. Installs a validating webhook with failurePolicy: Fail. Its Service has no endpoints, so every call fails, and the selector limits this to the create of chi-repro-default-0-1.
  3. Adds a label to the CHI. The label is copied to the StatefulSet volumeClaimTemplates, which makes the next update Forbidden.
  4. Restarts the operator. The reconcile recreates 0-0, deletes 0-1, and the create of 0-1 fails.
  5. Deletes the webhook and waits 120 seconds.
Expected

The operator retries the create after a failed reconcile, and the StatefulSet comes back once the webhook works again.

Actual
  • CHI status is Aborted.
  • status.hosts is 2, but only 1 StatefulSet exists.
  • 15 minutes after the webhook is deleted, the StatefulSet is still missing.
  • The operator logged no reconcileCR lines in the last 13 of those minutes.

Log:

E statefulset-reconciler.go:530] doUpdateStatefulSet():Host:0-1[0/1]:repro/repro:StatefulSet update failed. err: StatefulSet.apps "chi-repro-default-0-1" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ...
I statefulset-reconciler.go:337] updateStatefulSet():Host:0-1[0/1]:repro/repro:Update StatefulSet(repro/chi-repro-default-0-1) switch from Update to Recreate
E statefulset-reconciler.go:469] doCreateStatefulSet():Host:0-1[0/1]:repro/repro:StatefulSet create failed. err: Internal error occurred: failed calling webhook "sts-create.chop-repro.local": ...
E statefulset-reconciler.go:442] shouldAbortOrContinueCreateStatefulSet():Host:0-1[0/1]:repro/repro:Create StatefulSet: repro/chi-repro-default-0-1 - failed and needs recreate; propagating as error
W worker-reconciler-chi.go:989] first shard failed, skipping rest of shards due to an error: crud error - should recreate
W worker.go:482] markReconcileCompletedUnsuccessfully():CHI:repro/repro:reconcile completed UNSUCCESSFULLY, task id: ...

The PVC of the lost host stays, so the data is kept. The replica is down until someone acts.

Workaround

Restart the operator when a CHI is Aborted. A CronJob doing this brought the StatefulSet back in our test.

Lingua principale
Go
Stelle
2.6k
Fork
584
Merge medio
3g 11h
PR unite (30g)
7

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Altinity/clickhouse-operator

Tutte le issue di Altinity/clickhouse-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.