Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

ClickHouseKeeperInstallation stuck at status: InProgress despite healthy 3-node quorum (operator 0.27.0)

Open
  • Is something wrong?
#2,035 5 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
devops, infrastructure

Research direction

Start with worker-reconciler-chk.go, especially reconcileCR() and prepareStsReconcileOptsWaitSection(), then inspect object-status.go around GetObjectStatusFromMetas(). Compare the CHK reconciliation and status finalization path with the sibling CHI behavior described in the report. Done means a healthy three-replica Keeper reaches status Completed without repeated reconciliation or growing taskIDsStarted.

Written by the indexing model from the issue text.

Description

Keeper
Description

With altinity-clickhouse-operator 0.27.0, a ClickHouseKeeperInstallation (CHK) never transitions to status: Completed. It stays InProgress indefinitely even though the Keeper cluster is fully healthy, and the operator reconcile-loops on the CHK every ~10s without converging. A sibling ClickHouseInstallation (CHI) managed by the same operator reconciles to Completed normally.

Environment
  • clickhouse-operator: 0.27.0 (chart altinity-clickhouse-operator-0.27.0)
  • Kubernetes: 1.35 (GKE)
  • ClickHouse Keeper image: 26.5.x, 1 shard / 3 replicas
What's healthy

The Keeper has a full quorum — from mntr on a node:

zk_server_state        leader
zk_followers           2
zk_synced_followers    2

All 3 keeper pods are Running/Ready, the StatefulSets report rolling-update complete, and the ClickHouseInstallation (server) that uses this Keeper reaches status: Completed.

Observed behavior

The CHK stays status: InProgress; taskIDsStarted keeps growing while taskIDsCompleted stays at 0. The operator repeats this cycle continuously:

worker-reconciler-chk.go:57  reconcileCR():start
object-status.go:47          GetObjectStatusFromMetas(): cur and new objects are equal ... Update of the object is not required
worker-reconciler-chk.go:795 prepareStsReconcileOptsWaitSection(): Setting option SetWaitUntilReady   (once per host)
worker-reconciler-chk.go:91  reconcileCR(): CR has reconcile work - continue reconcile
worker-reconciler-chk.go:132 reconcileCR():end

It reproduces on a freshly created cluster, so it is not tied to any pre-existing or corrupted state.

Expected

Once the Keeper StatefulSets are ready and the quorum is healthy, the CHK should reach status: Completed (as the CHI does).

Questions
  • Is there a readiness/coordination gate (e.g. reconcile.coordination.keeper.readyTimeout) that the CHK controller waits on and never satisfies in this scenario?
  • Any known issue with CHK status finalization in 0.27.0?

Happy to add a minimal reproducer manifest and full operator logs.

Dominant language
Go
Stars
2.6k
Forks
579
Avg merge
5d 22h
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Altinity/clickhouse-operator

All issues in Altinity/clickhouse-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.