Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Lost-volume recovery drops nothing when the lost host is the shard's first host (provisioner: Operator)

Open
  • Is something wrong?
#2,096 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
go, kubernetes, sql
Domain
databases, devops

Research direction

Start in pkg/controller/chi/worker-deleter.go around lines 466-470, where dropZKReplica runs on shard.FirstHost(), then read the statements built in pkg/model/chi/schemer/sql.go lines 244-249. Next, check migrateTables in pkg/controller/chi/worker-migrator.go (lines 90 and 101-119), which swallows the REPLICA_ALREADY_EXISTS error. Done means a lost first-host volume recovers its database and table replicas from Keeper without manual intervention, with the drop running on a healthy peer or using explicit FROM ZKPATH paths.

Written by the indexing model from the issue text.

Description

Summary

With storageManagement.provisioner: Operator, when the volume of a shard's first host is lost, the operator's lost-volume recovery drops nothing: it runs the stale-replica drop on the shard's first host, which is the freshly empty host itself. The CREATE DATABASE it then replays fails with REPLICA_ALREADY_EXISTS, the error is logged and swallowed, the host is recorded in hostsWithTablesCreated, and the CHI reports Completed with the replica empty. Nothing retries it.

The same recovery works when the lost host is any other host of the shard.

Versions
  • clickhouse-operator 0.27.3 (default reconcile settings; host.drop.replicas.onLostVolume: yes)
  • ClickHouse 26.7.3.19
  • kind (local-path StorageClass), 1 shard × 3 replicas
Steps to reproduce
  1. CHI with spec.defaults.storageManagement.provisioner: Operator, 1 shard, 3 replicas, Keeper configured.
  2. Create a Replicated database and a table in it: 
    CREATE DATABASE db ON CLUSTER '{cluster}' ENGINE = Replicated('/clickhouse/databases/db', '{shard}', '{replica}');
CREATE TABLE db.t (id UInt64) ENGINE = ReplicatedMergeTree ORDER BY id;
INSERT INTO db.t SELECT number FROM numbers(10000);
  3. Lose the first host's volume: kubectl delete pvc default-chi-<chi>-<cluster>-0-0-0 --wait=false; kubectl delete pod chi-<chi>-<cluster>-0-0-0.
  4. Trigger a reconcile (e.g. patch spec.taskID; see the note below).
Expected

The stale 0|chi-...-0-0 database replica and the table's chi-...-0-0 replica are dropped, the database is recreated on 0-0, and it recovers its tables from Keeper.

Actual
  • The PVC and StatefulSet are recreated; the pod starts empty.
  • dropZKReplica runs on shard.FirstHost() (pkg/controller/chi/worker-deleter.go:466-470, "Sometimes host to drop is already unavailable, so let's run SQL statement of the first replica in the shard"). For the first host that is the host being recovered.
  • The statements are SYSTEM DROP REPLICA '<r>' and SYSTEM DROP DATABASE REPLICA '<shard>|<r>' with no FROM (pkg/model/chi/schemer/sql.go:244-249). Without FROM, ClickHouse iterates the local tables/databases, of which the empty host has none, so nothing is dropped.
  • CREATE DATABASE IF NOT EXISTS db ENGINE = Replicated(...) then fails: Code: 253 ... Replica ... already exists ... (REPLICA_ALREADY_EXISTS).
  • migrateTables ignores the drop result and logs the create error, then pushes the host to hostsWithTablesCreated and returns nil (pkg/controller/chi/worker-migrator.go:90, 101-119). The CHI reports Completed.
Suggested fix

Either run the drop on a healthy peer of the lost host (any other host of the shard), or keep running it on the lost host but name the paths explicitly, which ClickHouse allows on a server with no local table on them:

SYSTEM DROP REPLICA '<r>' FROM ZKPATH '<table zookeeper_path>';            -- path read from a peer's system.replicas
SYSTEM DROP DATABASE REPLICA '<shard>|<r>' FROM ZKPATH '<database zk path>';

We verified the second form recovers the replica completely when run on the empty first host itself.

Related (maybe a separate issue)

Nothing re-queues the CHI after the PVC is deleted: there is no PVC informer, the CHI resync only acts on spec changes, and recovery.onStatus.completed.onPodNotReady skips Pending pods (worker-status-helpers.go:123). The pod stays Pending ("persistentvolumeclaim not found") until something changes the spec. We work around it by patching spec.taskID from a sidecar.

Dominant language
Go
Stars
2.6k
Forks
579
Avg merge
5d 22h
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Altinity/clickhouse-operator

All issues in Altinity/clickhouse-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.