Adding a disk restarts ClickHouse in place before the pod has the new volume mount

Cerrado
  • ¿Algo no está bien?
#2,121 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Ya se ha fusionado un pull request relacionado.

  • #2122 de @lizardoluis — fusionado

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
clickhouse, go, kubernetes
Área
backend, devops

Línea de trabajo

Start in the Go reconciliation path at reconcileHostStatefulSet, then inspect shouldForceRestartHost and hostRequiresStatefulSetRollout to understand why a new volume mount does not defer the restart. Review the failing e2e test test_010036 and the related ordering issue #1926. Done means the StatefulSet rollout gives the pod its new mount before ClickHouse restarts, with the regression covered by a passing test.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

When a disk is added to a running ClickHouseInstallation, the operator restarts ClickHouse in place before the pod has the new volume mount. The new storage_configuration is already in the ConfigMap, so ClickHouse starts looking for a directory that does not exist in the pod yet. On ClickHouse 26.7+ that aborts startup and the pod crash-loops until the StatefulSet rollout catches up. On 26.3 the same sequence is tolerated, which is why it goes unnoticed there.

This is the same ordering problem as #1926 (ConfigMap updated, then SYSTEM SHUTDOWN, then the StatefulSet), but triggered by a volume mount instead of an image change.

Cause

In reconcileHostStatefulSet the config is updated, then shouldForceRestartHost triggers the in-place restart (hostSoftwareRestart), and only afterwards the StatefulSet is rolled out. The restart is skipped only when hostRequiresStatefulSetRollout returns true, and that function compares just the container Env. A new volumeMounts entry is not a reason to skip it.

To Reproduce
  1. Operator 0.27.4 (also present on 0.28.0 and master), ClickHouse 26.8, a CHI with storageManagement.provisioner: Operator and a default volumeClaimTemplate.
  2. Update the CHI to add a second volumeClaimTemplate, a volumeMounts entry for it in the pod template (/var/lib/clickhouse2), and storage_configuration/disks/disk2/path: /var/lib/clickhouse2/ in configuration.settings.
  3. The operator logs Host software restart start, the pod restarts, and ClickHouse exits with PATH_ACCESS_DENIED on /var/lib/clickhouse2. The pod goes into CrashLoopBackOff until the rollout recreates it.

The e2e test test_010036 fails this way on 26.8 and passes on 26.3.

Expected behavior

The operator rolls the pod out first, so the restarted ClickHouse already has the mount.

Environment
  • Operator 0.27.4
  • ClickHouse 26.8 (fails), 26.3 (passes)
  • Kubernetes 1.33 (minikube)
Additional context

ClickHouse 26.7+ treating the missing directory as fatal is a regression on the server side. I reported it at ClickHouse/ClickHouse#127938.

Lenguaje dominante
Go
Estrellas
2.6k
Forks
584
Merge medio
5 d 22 min
PR fusionados (30 d)
12

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Altinity/clickhouse-operator

Todos los issues de Altinity/clickhouse-operator

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.