Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

buildCR issues one uncached StatefulSet GET per host (twice per reconcile) instead of a single List/cache

Open
  • Is something wrong?
#2,063 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
go, kubernetes
Domain
infrastructure

Research direction

Start in pkg/controller/chi/kube/statesfulset.go, then trace buildCR through fillCurSTS and findMinMaxVersions to confirm where StatefulSet GETs are issued. Compare the informer/lister, labeled List, and cached-read options described in the issue, and preserve detection of genuinely absent StatefulSets. Done means the reconcile no longer performs repeated uncached per-host reads while retaining correct host status handling.

Written by the indexing model from the issue text.

Description

big deployment planned for review
Problem

On every reconcile, buildCR walks all hosts and reads each host's current StatefulSet individually:

  • fillCurSTS → STS().Get(ctx, host) for every host
  • findMinMaxVersions → PrepareHostStatefulSetWithStatus → getStatefulSetStatus → STS().Get(...) for every host

Each read is a live call to the API server (pkg/controller/chi/kube/statesfulset.go): kubeClient.AppsV1().StatefulSets(ns).Get(..., NewGetOptions()), where NewGetOptions() returns an empty metav1.GetOptions{} — i.e. no ResourceVersion, so it's a full read from etcd rather than a cached read. The host walk also runs twice per buildCR (once before pod-IP collection, once after).

So a single reconcile does on the order of 4 × hostCount sequential StatefulSet GETs before it reconciles the first host.

Impact

On a several-hundred-host CHI this "build" pre-pass is latency-bound and can take minutes on its own, before any host reconcile begins. It also adds etcd/apiserver read load that grows linearly with host count. Together with #2045, it contributes to large installations appearing to make little progress, since a good share of the wall-clock is spent re-reading state that could be fetched in one call or served from cache.

Fix ideas
  • Read StatefulSets from the informer/lister cache the operator already maintains, instead of the live clientset.
  • Or replace the per-host GETs with a single labeled List per namespace into a map, and look up per host (a miss then means genuinely absent, which still detects deleted StatefulSets correctly).
  • Alternatively, use a cached read (GetOptions{ResourceVersion: "0"}, served from the apiserver watch cache), which avoids the etcd round-trip per call.
Dominant language
Go
Stars
2.6k
Forks
577
Avg merge
1d 6h
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Altinity/clickhouse-operator

All issues in Altinity/clickhouse-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.