CHK reconciles in a tight loop after its first Completed; with `onKeeperResourceUpdate: reconcile` the dependent CHI never finishes (0.27.4)

Start with keeperPredicate in cmd/operator/app/thread_keeper.go, which passes every CHK update, then trace reconcileCR and HasReconcileWork in pkg/controller/chk/worker-reconciler-chk.go to see why a CHK with an empty action plan keeps continuing. The CHI side is shouldReconcileOnKeeperUpdate and enqueueDependentCHIs in pkg/controller/chi/controller-chk-watcher.go. Done means the CHK settles after its first Completed and a CHI with no NormalizedCRCompleted gets created and deleted without a reconcile livelock.

Environment

Item	Value
clickhouse-operator	0.27.4 (Helm chart altinity-clickhouse-operator 0.27.4, image tag = appVersion)
Kubernetes	Amazon EKS 1.35, platform version eks.24
ClickHouse server	clickhouse/clickhouse-server:26.8.25.14
ClickHouse Keeper	clickhouse/clickhouse-keeper:26.8.25.14
Operator scope	one namespace (clickhouse), namespaced RBAC
Operator config	reconcile.coordination.keeper.readyTimeout: 600, reconcile.coordination.keeper.onKeeperResourceUpdate: reconcile

Summary

We created a ClickHouseInstallation and a ClickHouseKeeperInstallation with the same name in one apply.
The CHI referenced the CHK with spec.configuration.zookeeper.keeper.name.

Two things went wrong at once.

1. The CHK reconciled in a tight loop. It reached Completed about 92 times a minute for two hours.
   Each cycle had an empty action plan but still logged CR has reconcile work - continue reconcile.
   The loop started 5 ms after the first Completed and stopped only when we deleted the CHK.
2. The CHK watcher turned that loop into a CHI livelock.
   Every CHK Completed enqueued a ReconcileCHI cmd=add for the dependent CHI.
   The CHI had no NormalizedCRCompleted yet, so shouldReconcileOnKeeperUpdate always returned true.
   The in-flight CHI reconcile was cancelled every time a new command arrived.
   The CHI never got past keeper resolution and never created its StatefulSet.
   kubectl delete chi was blocked the same way: deleteCHI() started 70 times a minute and never finished.

A production CHI and CHK in the same namespace were not affected. They were already fully reconciled.

Steps to reproduce

1. Run operator 0.27.4 with this config:

   reconcile:
     coordination:
       keeper:
         readyTimeout: 600
         onKeeperResourceUpdate: reconcile
   

2. Apply a CHK and a CHI together, in one kubectl apply (we used one Helm release):

   apiVersion: clickhouse-keeper.altinity.com/v1
   kind: ClickHouseKeeperInstallation
   metadata:
     name: restore
     namespace: clickhouse
   spec:
     defaults:
       storageManagement:
         provisioner: Operator
         reclaimPolicy: Retain
       templates:
         podTemplate: keeper
         dataVolumeClaimTemplate: keeper-data
     configuration:
       clusters:
         - name: keeper
           layout:
             replicasCount: 1
     templates:
       podTemplates:
         - name: keeper
           spec:
             containers:
               - name: clickhouse-keeper
                 image: clickhouse/clickhouse-keeper:26.8.25.14
       volumeClaimTemplates:
         - name: keeper-data
           spec:
             storageClassName: gp3
             accessModes: [ReadWriteOnce]
             resources:
               requests:
                 storage: 5Gi
   ---
   apiVersion: clickhouse.altinity.com/v1
   kind: ClickHouseInstallation
   metadata:
     name: restore
     namespace: clickhouse
   spec:
     configuration:
       zookeeper:
         keeper:
           name: restore
       clusters:
         - name: swaap
           layout:
             shardsCount: 1
             replicasCount: 1
     templates:
       podTemplates:
         - name: server
           spec:
             containers:
               - name: clickhouse
                 image: clickhouse/clickhouse-server:26.8.25.14
   

3. Watch the operator log. Within one minute the CHK completes, then completes again every 0.4 to 0.7 s.
   The CHI never creates chi-restore-swaap-0-0.

4. Run kubectl -n clickhouse delete chi restore. The delete never completes.

Observed behaviour

Rates (operator log, Loki count_over_time per minute)

Phase 1, 09:22:47Z to 10:03Z, CHI and CHK both present:

Log line	Per minute
CHK clickhouse/restore reached Completed — looking for dependent CHIs	92
No NormalizedCRCompleted for CHI clickhouse/restore — cannot diff, enqueueing	92
Triggering reconcile for CHI clickhouse/restore due to CHK clickhouse/restore completing	92
ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore	92
client rate limiter Wait returned an error: context canceled (CHI keeper resolver)	23 to 36
deleteCHI():CHI:clickhouse/restore:will delete CR with all resources (after 09:37:54Z)	70

Phase 2, 10:03Z to 11:26Z, CHI recreated with zookeeper.nodes and healthy, CHK still present:

Log line	Per minute
CHK clickhouse/restore reached Completed	92
any line mentioning the CHI	0

Totals for 2026-10-15: CHK restore ran reconcileCR() 11,427 times. The production CHK swaap in the same namespace ran it 24 times. The CHI restore ran reconcileCR() 530 times and never completed in phase 1.

Log excerpts

Creation and the first CHK completion. The CHK restarts its reconcile 5 ms after completing, before the watcher fires:

I1015 09:22:01.650178       1 controller.go:719] ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore
I1015 09:22:01.784951       1 worker-reconciler-chk.go:58] worker-reconciler-chk.go:58:reconcileCR():start:CHK:clickhouse/restore
I1015 09:22:47.156775       1 worker.go:336] finalizeReconcileAndMarkCompleted():CHK:clickhouse/restore:reconcile completed successfully, task id: auto-54df58ca-bede-450b-96b2-5573e1174969
I1015 09:22:47.157603       1 controller-chk-watcher.go:112] CHK clickhouse/restore reached Completed — looking for dependent CHIs
I1015 09:22:47.162484       1 worker-reconciler-chk.go:58] worker-reconciler-chk.go:58:reconcileCR():start:CHK:clickhouse/restore
I1015 09:22:47.503048       1 worker.go:336] finalizeReconcileAndMarkCompleted():CHK:clickhouse/restore:reconcile completed successfully, task id: auto-222ea6fe-04d4-4980-addc-ef9497a93701
I1015 09:22:47.559295       1 controller-chk-watcher.go:221] CHI:clickhouse/restore:No NormalizedCRCompleted for CHI clickhouse/restore — cannot diff, enqueueing
I1015 09:22:47.559317       1 controller-chk-watcher.go:197] Triggering reconcile for CHI clickhouse/restore due to CHK clickhouse/restore completing
I1015 09:22:47.560022       1 controller.go:719] ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore

One CHK cycle in steady state. The action plan line is empty, yet the reconcile continues and rewrites every object:

I1015 10:30:00.437388       1 worker.go:336] finalizeReconcileAndMarkCompleted():CHK:clickhouse/restore:reconcile completed successfully, task id: auto-21535e60-ee0f-4df2-9227-16dea3453183
I1015 10:30:00.444653       1 worker-reconciler-chk.go:58] worker-reconciler-chk.go:58:reconcileCR():start:CHK:clickhouse/restore
I1015 10:30:00.444688       1 worker.go:483] createTemplatedCR():CHK:clickhouse/restore:CR has an ancestor, use it as a base for reconcile. CR: clickhouse/restore
I1015 10:30:00.476445       1 worker-reconciler-chk.go:203] CHK:clickhouse/restore:
I1015 10:30:00.476482       1 worker-reconciler-chk.go:110] reconcileCR():CHK:clickhouse/restore:CR has reconcile work - continue reconcile
I1015 10:30:00.506139       1 worker-config-map.go:70] updateConfigMap():CHK:clickhouse/restore:Update ConfigMap clickhouse/chk-restore-common-configd
I1015 10:30:00.596622       1 worker-pdb.go:36] PDB updated: clickhouse/chk-restore-keeper
I1015 10:30:00.711945       1 worker-service.go:206] updateService():CHK:clickhouse/restore:Update Service success: clickhouse/chk-restore-keeper-0-0
I1015 10:30:00.800383       1 worker.go:190] shouldForceRestartHost():Host:0-0[0/0]:clickhouse/restore:Host force restart is not required. Host: 0-0
I1015 10:30:00.909489       1 worker-service.go:206] updateService():CHK:clickhouse/restore:Update Service success: clickhouse/keeper-restore
I1015 10:30:01.087557       1 controller-chk-watcher.go:112] CHK clickhouse/restore reached Completed — looking for dependent CHIs

The CHI side during phase 1. The reconcile restarts about every 0.4 to 1.4 s and the keeper resolver sees a cancelled context:

I1015 09:30:00.162374       1 worker.go:384] worker.go:347:updateCHI():end:CHI:clickhouse/restore
I1015 09:30:00.162439       1 worker.go:346] worker.go:346:updateCHI():start:CHI:clickhouse/restore
I1015 09:30:00.162476       1 worker.go:354] updateCHI():CHI:clickhouse/restore:finalizer in place, proceed to reconcile cycle. CHI: clickhouse/restore
I1015 09:30:00.361301       1 controller.go:719] ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore
I1015 09:30:01.167511       1 controller-keeper-resolver.go:82] Keeper replica discovery failed or returned 0 nodes, falling back to CR service: failed to resolve keeper ref: host services clickhouse/restore: client rate limiter Wait returned an error: context canceled

kubectl delete chi restore at 09:37:54Z. The delete starts, the watcher enqueues another add, and the delete starts again one second later:

I1015 09:37:54.163551       1 worker-deleter.go:765] deleteCHI():CHI:clickhouse/restore:will delete CR with all resources: clickhouse/restore
I1015 09:37:54.181214       1 worker-deleter.go:291] deleteCRProtocol():CHI:clickhouse/restore:Delete CHI started
I1015 09:37:54.362168       1 controller-chk-watcher.go:221] CHI:clickhouse/restore:No NormalizedCRCompleted for CHI clickhouse/restore — cannot diff, enqueueing
I1015 09:37:54.363173       1 controller.go:719] ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore
I1015 09:37:55.362075       1 worker-deleter.go:765] deleteCHI():CHI:clickhouse/restore:will delete CR with all resources: clickhouse/restore

Recovery at 10:02:56Z. We removed the CHI finalizer by patch, deleted the chi-storage-restore ConfigMap, and recreated the CHI with zookeeper.nodes. The CHI then completed in about 90 s. The CHK kept looping:

I1015 10:02:56.009796       1 controller.go:719] ENQUEUE new ReconcileCHI cmd=add for clickhouse/restore
I1015 10:02:56.056575       1 worker.go:602] createTemplatedCR():CHI:clickhouse/restore:CR has NO ancestor, use empty base for reconcile. CR: clickhouse/restore
I1015 10:03:01.653536       1 worker-service.go:217] createService():CHI:clickhouse/restore:OK Create Service: clickhouse/chi-restore-swaap-0-0
I1015 10:04:27.708233       1 worker-reconciler-chi.go:239] CHI:clickhouse/restore:IPs of the CR clickhouse/restore: len: 1 [10.50.77.65]

Code path (release-0.27.4)

• pkg/controller/chi/controller-chk-watcher.go
  • onCHKUpdate fires on every CHK update whose status.status moves to Completed.
  • enqueueDependentCHIs lists CHIs, keeps those whose zookeeper.keeper matches the CHK, and calls shouldReconcileOnKeeperUpdate.
  • shouldReconcileOnKeeperUpdate returns true when cr.GetAncestorT() is nil ("No NormalizedCRCompleted ... cannot diff, enqueueing"). A CHI that has never completed always hits this branch.
  • It then enqueues cmd_queue.NewReconcileCHI(cmd_queue.ReconcileAdd, nil, chi). The comment says ReconcileAdd is used because prepareCHIAdd returns true unconditionally.
• cmd/operator/app/thread_keeper.go
  • The CHK controller is built with For(&api.ClickHouseKeeperInstallation{}, builder.WithPredicates(keeperPredicate())) and Owns(&apps.StatefulSet{}).
  • keeperPredicate().UpdateFunc returns true for every update. It does not filter on generation. Status writes therefore re-enqueue the CHK.
• pkg/controller/chk/controller.go calls w.reconcileCR(ctx, nil, new) with old == nil, so isGenerationTheSame(old, new) cannot short-circuit.
• pkg/controller/chk/worker-reconciler-chk.go
  • reconcileCR continues when new.HasReconcileWork() is true. In our logs this was true on every cycle, with an empty action plan.
  • Each cycle calls markReconcileStart, per-host updateCRObjectStatus, and finalizeReconcileAndMarkCompleted. Each of those is a CHK update event, so the status round trip feeds the next reconcile.
• pkg/controller/chi/controller-keeper-resolver.go:82 is where the cancelled CHI reconcile surfaces (Keeper replica discovery failed ... context canceled).

We could not tell from logs why HasReconcileWork() stays true for this CHK and not for our production CHK in the same namespace. Both run the same operator and chart. Differences: the looping CHK has one replica, and it was created in the same apply as its CHI.

Expected behaviour

• A CHK with an empty action plan should log No reconcile work - abort reconcile and stop. It should not reach Completed 92 times a minute.
• One CHK completion should trigger at most one CHI reconcile, and that reconcile should be allowed to finish. The watcher should not enqueue a new add while a reconcile for the same CHI is still running, or it should coalesce.
• kubectl delete chi should complete even while the watcher is active.

Workaround

Do not reference the CHK from the CHI. Point the CHI at the CHK client Service instead:

spec:
  configuration:
    zookeeper:
      nodes:
        - host: keeper-restore
          port: 2181

With this the CHI is no longer a "dependent CHI", the watcher skips it, and the CHI completes. The CHK loop itself continues until the CHK is deleted.

To unblock a stuck delete: patch the CHI finalizers to [], delete the chi-storage-<name> ConfigMap, then delete the CHI.

Secondary: CRD lookup needs cluster-scoped RBAC

With the chart's namespaced RBAC, every CHI and CHK delete logs this and continues:

E1015 11:26:32.520397       1 worker-deleter.go:764] deleteCHI():CHI:clickhouse/restore:unable to get CRD, got error: customresourcedefinitions.apiextensions.k8s.io "clickhouseinstallations.clickhouse.altinity.com" is forbidden: User "system:serviceaccount:clickhouse:clickhouse-operator" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
E1015 11:26:32.792197       1 worker-deleter.go:277] deleteCHK():CHK:clickhouse/restore:unable to get CRD, got error: customresourcedefinitions.apiextensions.k8s.io "clickhousekeeperinstallations.clickhouse-keeper.altinity.com" is forbidden: User "system:serviceaccount:clickhouse:clickhouse-operator" cannot get resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

customresourcedefinitions is cluster-scoped, so a namespaced Role can never grant it. We added a ClusterRole with get on customresourcedefinitions. The chart could ship that ClusterRole when rbac.namespaceScoped is set, or the deleter could treat the forbidden error as "CRD present".

Related issues

• #1913 Race condition: deploying CHI after Keeper leaves CHI unconnected to Keeper/replication (closed). Same CHI and CHK ordering, different symptom.
• #2035 ClickHouseKeeperInstallation stuck at status InProgress despite healthy 3-node quorum (open).

No release after 0.27.4 exists as of 2026-10-16. pkg/controller/chi/controller-chk-watcher.go on master is byte-identical to the 0.27.4 tag (blob 5c9100df). No merged PR since 2026-09-24 mentions the keeper watcher or onKeeperResourceUpdate.

---