High availability for the Klio Server (Tier 1): avoid WAL accumulation on the primary when Tier 1 is unavailable

オープン
  • 何か問題がありますか？
#513 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
go, kubernetes, postgresql
領域
databases, devops, infrastructure

調査の方向性

Start by reading the documentation sections on Managing Storage and the Server custom resource, then trace how the klio physical replication slot is created and advanced. The issue proposes several different directions, from HA to safeguards, documentation, and metrics, so agree on a specific scope before starting; done means implementing that scope and documenting the behavior and recovery steps.

索引モデルが issue の本文から書いたものです。

説明

Summary

The Klio Server (Tier 1) currently runs as a single-replica StatefulSet backed by a PVC.
Because WAL is streamed through the physical replication slot klio, any prolonged
unavailability of Tier 1 makes PostgreSQL retain WAL on the primary indefinitely.
A backup-infrastructure outage can therefore turn into a database outage once the
primary's WAL volume fills up.

We'd like to request an HA story for Tier 1, or at least documented failure semantics
and built-in safeguards.

Context

  • Klio version: 0.0.20
  • CloudNativePG on RKE2, PostgreSQL 18
  • PostgreSQL data/WAL on node-local storage (TopoLVM), which is a common setup for CNPG
  • Two data centers, no stretched storage

As far as we can tell from the documentation:

  • The Server CR creates a StatefulSet with a single server container
    (pod <name>-klio-0); replicas, failover and leader election are not mentioned.
  • The klio-plugin sidecar creates the physical slot klio on the primary.
  • "Managing Storage" states that when Tier 1 is full, WAL streaming stops and
    "PostgreSQL accumulates WAL on its own PVC".
  • With node-local storage for Tier 1, the server pod cannot be rescheduled to another
    node while its node is down.

Problem

If the Klio Server pod or its node is unavailable for longer than the WAL headroom on
the primary allows (node failure, storage issue, Tier 1 PVC full, misconfiguration):

  1. The klio slot stops advancing.
  2. The primary retains all WAL since the slot's restart_lsn.
  3. The WAL PVC fills up and the primary stops accepting writes.

When synchronous mode is enabled, an unavailable Tier 1 could block commits even
earlier, unless quorum with other standbys is configured.

Today the only safeguards are on the user side:

  • max_slot_wal_keep_size, which deliberately breaks WAL continuity (the slot gets
    invalidated, leaving a gap in the archive and requiring a new base backup),
  • oversizing the WAL PVCs, auto-expanding them, and alerting.

Proposals (any of these would help)

  1. Multi-replica / active-passive Klio Server: e.g. a standby replica with leader
    election that can take over WAL streaming, with its own or replicated storage.
  2. Degraded mode / fallback: when Tier 1 is unreachable for a configurable time,
    let the sidecar ship WAL directly to Tier 2 (object storage), so the slot keeps
    advancing and WAL continuity is preserved.
  3. Built-in safeguard: an optional plugin-level threshold (retained WAL bytes or
    time) after which Klio takes a documented action, with clear events and metrics,
    instead of relying only on max_slot_wal_keep_size.
  4. Documentation of failure semantics, in particular:
    • behavior when the Klio Server is down for a long time,
    • how and when the klio slot is advanced ("controlled replication slot advancement"),
    • what happens to the klio slot on CNPG failover or switchover,
    • recovery procedure after the slot was invalidated (wal_status = 'lost'): does
      Klio detect it and trigger or require a new base backup?
    • guidance for running Tier 1 in multi-node / multi-AZ setups.
  5. Metrics and alerts: retained WAL behind the klio slot, streaming
    connected/disconnected state, and Tier 1 PVC usage (the docs note that there are
    no built-in storage alerts), plus example PrometheusRules.

Why this matters

Klio's design (WAL streaming, deduplication, tiering) is very attractive, but for
production adoption the backup path must not become a single point of failure for the
database itself. With archive_command-based solutions the same risk exists, but
operators are familiar with it. With Klio, Tier 1 adds an in-cluster component whose
availability directly affects the primary.

We're happy to test proposed solutions and share results.

主要言語
Go
スター
40
フォーク
10
平均マージ
19時間 58分
マージ済み PR（30日）
175

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

cloudnative-pg/klio のほかの issue

cloudnative-pg/klio の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。