Operator stalls after Velero restore; CHI stuck InProgress until operator pod restart — DeltaFIFO “slow event handlers” / high queue depth

Aperta
  • Qualcosa non va?
#1,969 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Bug
Chiarezza
Da chiarire
Stato di attività
Tranquilla
Stack tecnologico
go, kubernetes
Ambito
devops, infrastructure

Direzione di ricerca

Inizia riproducendo l’ordine di ripristino di Velero e l’errore transitorio di RBAC, quindi esamina il percorso di riconciliazione insieme a worker-config-map.go e alle tracce di informer/DeltaFIFO in trace.go. Raccogli i log completi richiesti dell’operatore, i dump delle goroutine e i manifesti CHI/STS; il lavoro è completato quando viene identificata la causa del blocco e viene dimostrato il recupero senza riavviare l’operatore.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

After a Velero-based restore (backup/restore), the ClickHouse operator sometimes stops making progress on a ClickHouseInstallation (CHI): the CHI remains InProgress with HOSTS-COMPLETED empty even though the ClickHouse pod is Running. Operator logs stop advancing. Restarting the operator pod restores normal reconciliation.

We also see Kubernetes client informer traces indicating slow event handlers and non-trivial DeltaFIFO depth (~51), which suggests the operator’s shared work queue may be backing up or wedged after the restore storm / transient RBAC failures.

We are reporting this upstream in case it is a known class of issues (informer backlog, error handling after Forbidden/RBAC, or reconciliation blocking the shared queue).

Steps to reproduce
  1. Take a backup with Velero
  2. Restore the backup into a target cluster/namespace.
  3. Observe restore ordering:
    • Operator pod comes up before ServiceAccount / Role / RoleBinding for the operator in some cases (delay < ~1 minute).
  4. During that window, the operator logs authentication/authorization errors (cannot list/watch CHI, STS, PVC, etc.).
  5. After RBAC objects appear, errors stop and the ClickHouse pod from the CHI can become Running.
  6. Bug: The CHI STATUS stays InProgress; HOSTS-COMPLETED does not advance; no further meaningful operator logs for the reconciliation path; operator appears stalled.
  7. Workaround: Delete/restart the operator pod; reconciliation resumes and the CHI completes as expected.

Again:

  1. In Velero, changed ordering in the restore to create role,rolebinding,sa first.
  2. The operator came up and role/rolebinding/sa was created beforehand and didn't notice the RABC issue.
  3. However, the Operator appears stalled, CHI stays InProgress
Expected behavior
  • After transient RBAC or API errors during restore, the operator should recover and eventually mark the CHI complete without requiring a manual operator restart.
  • If the work queue is temporarily overloaded, we would still expect progress or clear error/retry signals, not a silent stall with a stuck CHI status.
Actual behavior
  • CHI example while stuck:

NAME STATUS CLUSTERS HOSTS HOSTS-COMPLETED AGE clickhouse InProgress 1 1 ~10m+

  • ClickHouse pod state (example): chi-clickhouse-default-0-0-0 1/1 Running.
  • Operator pod: 2/2 Running, but no further useful logs after the stall begins.
Operator logs
I0424 06:13:50.985141       1 worker-config-map.go:70] updateConfigMap():CHI:default/clickhouse:Update ConfigMap default/chi-clickhouse-common-usersd
I0424 06:13:58.594193       1 trace.go:236] Trace[1021980655]: "DeltaFIFO Pop Process" ID:default/temporal-frontend-headless-mqr5w,Depth:51,Reason:slow event handlers blocking the queue (24-Apr-2026 06:13:58.491) (total time: 102ms):
Trace[1021980655]: [102.62776ms] [102.62776ms] END

I0424 06:27:58.786132       1 trace.go:236] Trace[2079076590]: "DeltaFIFO Pop Process" ID:default/kube-state-metrics,Depth:51,Reason:slow event handlers blocking the queue (24-Apr-2026 06:27:58.683) (total time: 101ms):
Trace[2079076590]: [101.600123ms] [101.600123ms] END
I0424 06:27:58.887442       1 trace.go:236] Trace[580084308]: "DeltaFIFO Pop Process" ID:default/tia-plus.app-change-pmgcv,Depth:23,Reason:slow event handlers blocking the queue (24-Apr-2026 06:27:58.786) (total time: 100ms):
Trace[580084308]: [1

Every 2.0s: kubectl get chi; kubectl  get po | grep clickhouse

NAME                 STATUS       CLUSTERS   HOSTS   HOSTS-COMPLETED   AGE   SUSPEND
clickhouse.       InProgress               1          1                         10m

chi-clickhouse-default-0-0-0                        1/1     Running            0             10m
clickhouse-operator-774bbf94dd-vkns9     2/2     Running            0             15m
Workaround

Restart the clickhouse-operator deployment pod (or rollout restart). After restart, the CHI reconciles and completes.

Ask
  1. Is this a known interaction with Velero restore ordering / transient RBAC?
  2. Any recommended settings (workers, resync, timeouts) or version that mitigates informer backlog?
  3. If this matches a bug on your side, we are happy to collect goroutine dumps, full operator logs, or CHI/STS manifests as you specify.
Lingua principale
Go
Stelle
2.6k
Fork
584
Merge medio
3g 6h
PR unite (30g)
6

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Altinity/clickhouse-operator

Tutte le issue di Altinity/clickhouse-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.