First-class support (or an officially supported recipe) for multi-Kubernetes / multi-region DR
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 18/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- go, kubernetes
- Domain
- documentation, infrastructure
Research direction
This is a design question, not a code change yet. Start with the recovery-architecture doc linked in the issue and the CHI settings it mentions, zookeeper.nodes and remote_servers, to see what the operator already supports across clusters. A first useful step is a scoping discussion with maintainers on whether a docs and test checklist for two-CHI primary/warm-standby is accepted. Done means an agreed scope, not an implementation.
Written by the indexing model from the issue text.
Description
Summary
We run ClickHouse via clickhouse-operator (currently looking at 0.27.x) and need a clear story for disaster recovery across Kubernetes clusters, often in different regions: primary site + warm standby (or equivalent), with a tested failover path.
Today the operator is excellent inside a single Kubernetes cluster, but multi-site DR still looks like DIY. We’d like this to become a first-class, documented, and tested capability — or at least an explicitly supported topology with a maintained guide tied to operator releases.
What we need
Something teams can rely on in production, for example:
- Documented supported topology for primary / warm-standby (or another Altinity-recommended shape) across two Kubernetes clusters / regions, including Keeper placement (voting members vs observers), naming/
{replica}rules, and failover / failback steps. - Operator-level help where it makes sense — even if a full “Federated CHI” is a bigger project. Examples: clearer cross-site guidance in-tree, validation/warnings, helpers for
remote_servers/ macros, or a thin multi-cluster controller later. - Soak / failover testing so this isn’t only a blog/webinar recipe.
We’re fine doing networking ourselves (peering, DNS, service exposure). The gap is knowing what’s supported vs best-effort, and not inventing a one-off design that breaks on the next operator upgrade.
What we already understand (not asking for this)
- CHI is namespaced and the operator reconciles inside one Kubernetes cluster — that matches what we’ve seen in the CRD and in your materials.
- Multiple entries in
spec.configuration.clusters[]in one CHI are a same-cluster layout, not multi-region DR. - A real CHK is one Raft ensemble; we won’t treat extra CHK
clusters[]examples as multi-site Keeper HA. - Cross-region setups are possible today with two operators/CHIs, shared Keeper via
zookeeper.nodes, and a lot of network/DNS work (as in your blog and #1380).
Your DR webinar also called this out directly:
The operator operates within a single Kubernetes cluster, so multi-cluster deployments require something sitting outside the operator itself. […] eBay […] implemented a “Federated ClickHouseInstallation” […] We did not build it yet because the multi-region DR scenario is rare enough that it has not risen to the top of the priority list. But we are totally open to collaborating on it.
We’re essentially asking to raise the priority of that gap (docs + tests first is already valuable; federation later if it makes sense).
References
- https://docs.altinity.com/operationsguide/availability-and-recovery/recovery-architecture/
- https://altinity.com/blog/setting-up-cross-region-clickhouse-replication-in-kubernetes
- https://altinity.com/webinarspage/clickhouse-disaster-recovery-tips-and-tricks-to-avoid-trouble-in-paradise
- https://github.com/Altinity/clickhouse-operator/issues/1380
- https://github.com/Altinity/clickhouse-operator/issues/1258
Ask
- Is multi-Kubernetes / multi-region DR on the operator roadmap?
- Short of a federated controller, would you accept / help maintain an official in-repo guide + test checklist for the primary/warm-standby pattern with two CHIs?
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 579
- Avg merge
- 21h 2m
- Merged PRs (30d)
- 4
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Altinity/clickhouse-operator
-
Counter metrics are missing the conventional _total suffixPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Altinity/clickhouse-operator#2093 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Altinity/clickhouse-operator#2098 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Altinity/clickhouse-operator#2096 ·
Maintainers usually reply within 1 day
-
CHI reconcile aborts (and is never retried) when the API server briefly rejects owner-referenced objects right after the CRD is created (OpenShift OwnerReferencesPermissionEnforcement)Possibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 3/5 1-2 days Newbie friendliness 72/100
Altinity/clickhouse-operator#2094 ·
Maintainers usually reply within 1 day
-
[Regression / Discussion] Loss of hot-reloaded password rotation after removal of k8s_secret_* in 0.27.4Possibly taken @sunsingerus claimed this 9 days ago. Openplanned for review
Difficulty 5/5 Over a week Newbie friendliness 38/100
Altinity/clickhouse-operator#2092 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
All issues in Altinity/clickhouse-operator
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
[Bug]: Bedrock invoke-with-response-stream reports `stop_reason: "end_turn"` for tool-use turnsOpen
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
oalders/clodhopper#162 ·
Maintainers usually reply within 1 day
-
component/server severity/minor type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day