Adding a disk restarts ClickHouse in place before the pod has the new volume mount

Chiusa
  • Qualcosa non va?
#2,121 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Una pull request collegata è già stata integrata.

  • #2122 di @lizardoluis — integrata

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
clickhouse, go, kubernetes
Ambito
backend, devops

Direzione di ricerca

Start in the Go reconciliation path at reconcileHostStatefulSet, then inspect shouldForceRestartHost and hostRequiresStatefulSetRollout to understand why a new volume mount does not defer the restart. Review the failing e2e test test_010036 and the related ordering issue #1926. Done means the StatefulSet rollout gives the pod its new mount before ClickHouse restarts, with the regression covered by a passing test.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

When a disk is added to a running ClickHouseInstallation, the operator restarts ClickHouse in place before the pod has the new volume mount. The new storage_configuration is already in the ConfigMap, so ClickHouse starts looking for a directory that does not exist in the pod yet. On ClickHouse 26.7+ that aborts startup and the pod crash-loops until the StatefulSet rollout catches up. On 26.3 the same sequence is tolerated, which is why it goes unnoticed there.

This is the same ordering problem as #1926 (ConfigMap updated, then SYSTEM SHUTDOWN, then the StatefulSet), but triggered by a volume mount instead of an image change.

Cause

In reconcileHostStatefulSet the config is updated, then shouldForceRestartHost triggers the in-place restart (hostSoftwareRestart), and only afterwards the StatefulSet is rolled out. The restart is skipped only when hostRequiresStatefulSetRollout returns true, and that function compares just the container Env. A new volumeMounts entry is not a reason to skip it.

To Reproduce
  1. Operator 0.27.4 (also present on 0.28.0 and master), ClickHouse 26.8, a CHI with storageManagement.provisioner: Operator and a default volumeClaimTemplate.
  2. Update the CHI to add a second volumeClaimTemplate, a volumeMounts entry for it in the pod template (/var/lib/clickhouse2), and storage_configuration/disks/disk2/path: /var/lib/clickhouse2/ in configuration.settings.
  3. The operator logs Host software restart start, the pod restarts, and ClickHouse exits with PATH_ACCESS_DENIED on /var/lib/clickhouse2. The pod goes into CrashLoopBackOff until the rollout recreates it.

The e2e test test_010036 fails this way on 26.8 and passes on 26.3.

Expected behavior

The operator rolls the pod out first, so the restarted ClickHouse already has the mount.

Environment
  • Operator 0.27.4
  • ClickHouse 26.8 (fails), 26.3 (passes)
  • Kubernetes 1.33 (minikube)
Additional context

ClickHouse 26.7+ treating the missing directory as fatal is a regression on the server side. I reported it at ClickHouse/ClickHouse#127938.

Lingua principale
Go
Stelle
2.6k
Fork
584
Merge medio
5g 10h
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Altinity/clickhouse-operator

Tutte le issue di Altinity/clickhouse-operator

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.