Liveness probe failed

Start with the supplied ClickHouseKeeperInstallation YAML and the pod Events output, focusing on the liveness probe's connection to 127.0.0.1:2181 and the Keeper startup state. Check the operator and Keeper logs to determine why the probe cannot connect; done means documenting a confirmed cause and a reproducible configuration or operational fix.

Keeper pods are restarting with Liveness probe failed, I am using the altinity operator for keeper and clickhouse.

I have 3 keepers

this is my yaml manifest

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: ${keeper_metadata_name}
  namespace: ${namespace}
spec:
  configuration:
    clusters:
      - name: ${keeper_cluster_name}
        layout:
          replicasCount: ${keeper_replica_count}
    settings:
      logger/level: "information"
      logger/console: "true"
      listen_host: "0.0.0.0"
      keeper_server/four_letter_word_white_list: "*"
      keeper_server/coordination_settings/raft_logs_level: "information"
      prometheus/endpoint: "/metrics"
      prometheus/port: "7000"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "false"
      keeper_server/tcp_port: 2181

  defaults:
    templates:
      # Templates are specified as default for all clusters
      podTemplate: default
      dataVolumeClaimTemplate: default

  templates:
    podTemplates:
      - name: default
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          # affinity, remove it to allow use in single node test environment
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: topology.kubernetes.io/zone
                        operator: In
                        values:
%{ for zone in zones ~}
                          - ${zone}
%{ endfor ~}
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-keeper
                  topologyKey: "kubernetes.io/zone"
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: ${keeper_image_name}
              resources:
                requests:
                  memory: "256M"
                  cpu: "1"
                limits:
                  memory: "4Gi"
                  cpu: "2"
          securityContext:
            fsGroup: 101

          nodeSelector:
            role: "${label}"

    volumeClaimTemplates:
      - name: default
        reclaimPolicy: Retain
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: ${keeper_disk_size}
          storageClassName: ${storage_class_keeper}

failing with error

Events:                                                                                                                                                                                                                      │
│   Type     Reason     Age   From               Message                                                                                                                                                                       │
│   ----     ------     ----  ----               -------                                                                                                                                                                       │
│   Normal   Scheduled  62s   default-scheduler  Successfully assigned clickhouse/chk-agents-primary-0-0-0 to aks-chiagentpool-25729127-vmss000001                                                                             │
│   Normal   Pulled     55s   kubelet            Container image "clickhouse/clickhouse-keeper:latest" already present on machine                                                                                              │
│   Normal   Created    55s   kubelet            Created container clickhouse-keeper                                                                                                                                           │
│   Normal   Started    55s   kubelet            Started container clickhouse-keeper                                                                                                                                           │
│   Warning  Unhealthy  50s   kubelet            Liveness probe failed: Thu Aug  7 05:17:23 UTC 2025                                                                                                                           │
│ + date                                                                                                                                                                                                                       │
│ ++ exec                                                                                                                                                                                                                      │
│ bash: connect: Connection refused                                                                                                                                                                                            │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused                                                                                                                                                                    │
│ ++ printf ruok                                                                                                                                                                                                               │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ IFS=                                                                                                                                                                                                                      │
│ ++ tee                                                                                                                                                                                                                       │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ exec                                                                                                                                                                                                                      │
│ + OK=                                                                                                                                                                                                                        │
│ + [[ '' == \i\m\o\k ]]                                                                                                                                                                                                       │
│ + exit 1                                                                                                                                                                                                                     │
│   Warning  Unhealthy  45s  kubelet  Liveness probe failed: Thu Aug  7 05:17:28 UTC 2025                                                                                                                                      │
│ + date                                                                                                                                                                                                                       │
│ ++ exec                                                                                                                                                                                                                      │
│ bash: connect: Connection refused                                                                                                                                                                                            │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused                                                                                                                                                                    │
│ ++ printf ruok                                                                                                                                                                                                               │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ IFS=                                                                                                                                                                                                                      │
│ ++ tee