Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Regression / Discussion] Loss of hot-reloaded password rotation after removal of k8s_secret_* in 0.27.4

Open
  • Is something wrong?
#2,092 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 2 days

@sunsingerus is already working on this.

Since Oct 5, 2026.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes
Domain
databases, devops, security

Research direction

Start with docs/security_hardening.md and the ClickHouseInstallation user configuration path that handles valueFrom/secretKeyRef. Reproduce secret rotation followed by reconciliation, checking whether generated files under /etc/clickhouse-server/users.d/ change without restarting pods. Done means the supported rotation behavior is documented or the proposed zero-downtime mechanism is implemented and verified.

Written by the indexing model from the issue text.

Description

planned for review
Description

In release 0.27.4, support for the k8s_secret_* syntax was removed in favor of standard valueFrom / secretKeyRef references in ClickHouseInstallation user configurations (see Altinity ClickHouse Operator Security Hardening Documentation).

While migrating to secretKeyRef enforces namespace isolation and proper RBAC, it introduces an operational regression regarding password rotation and zero-downtime hot-reloading.

Regression Details

  1. Environment Variable Immutability:
    With valueFrom / secretKeyRef, secret values are passed into the clickhouse-server container as environment variables. The operator then renders XML references using the from_env attribute:

    <password from_env="CONFIGURATION_USERS_ADMIN_PASSWORD"></password>

    Since Kubernetes environment variables cannot be updated dynamically inside a running container, changing the underlying Kubernetes Secret value has no effect on running ClickHouse processes.

  2. Broken Password Rotation Workflow:

    • Previous Behavior (< 0.27.4):
      1. Update the password in the Kubernetes Secret.
      2. Trigger CHI reconciliation (e.g., updating a taskID annotation).
      3. The operator updated the mounted ConfigMap with the new password/hash.
      4. ClickHouse automatically hot-reloaded the updated XML files from disk without restarting pods.
    • Current Behavior (0.27.4):
      1. Updating the Kubernetes Secret and triggering reconciliation leaves the XML unchanged, as it only references static environment variable names.
      2. Tested and confirmed: The updated secret value is not picked up until pods are restarted.
      3. Impact: Rotating a password now requires a full rollout restart of all clickhouse-server pods.
Questions & Feature Suggestions
  1. Expected Rotation Strategy: Is requiring a full pod rollout restart the intended behavior for password rotation moving forward, or is there a supported method to achieve zero-downtime credential updates?
  2. Proposal (Secret Volume Mounts): Why not support mounting Kubernetes Secrets directly as files into /etc/clickhouse-server/users.d/ (or a mounted secrets directory) instead of injecting them as environment variables?
    • Mounting secrets directly would maintain proper RBAC and namespace security while restoring ClickHouse's native file-watching hot-reload capabilities.
Dominant language
Go
Stars
2.6k
Forks
577
Avg merge
1d 6h
Merged PRs (30d)
4

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Altinity/clickhouse-operator

All issues in Altinity/clickhouse-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.