High availability for the Klio Server (Tier 1): avoid WAL accumulation on the primary when Tier 1 is unavailable

未关闭
  • 哪里不对？
#513 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
go, kubernetes, postgresql
领域
databases, devops, infrastructure

调研方向

Start by reading the documentation sections on Managing Storage and the Server custom resource, then trace how the klio physical replication slot is created and advanced. The issue proposes several different directions, from HA to safeguards, documentation, and metrics, so agree on a specific scope before starting; done means implementing that scope and documenting the behavior and recovery steps.

由索引模型根据 Issue 内容生成。

描述

Summary

The Klio Server (Tier 1) currently runs as a single-replica StatefulSet backed by a PVC.
Because WAL is streamed through the physical replication slot klio, any prolonged
unavailability of Tier 1 makes PostgreSQL retain WAL on the primary indefinitely.
A backup-infrastructure outage can therefore turn into a database outage once the
primary's WAL volume fills up.

We'd like to request an HA story for Tier 1, or at least documented failure semantics
and built-in safeguards.

Context

  • Klio version: 0.0.20
  • CloudNativePG on RKE2, PostgreSQL 18
  • PostgreSQL data/WAL on node-local storage (TopoLVM), which is a common setup for CNPG
  • Two data centers, no stretched storage

As far as we can tell from the documentation:

  • The Server CR creates a StatefulSet with a single server container
    (pod <name>-klio-0); replicas, failover and leader election are not mentioned.
  • The klio-plugin sidecar creates the physical slot klio on the primary.
  • "Managing Storage" states that when Tier 1 is full, WAL streaming stops and
    "PostgreSQL accumulates WAL on its own PVC".
  • With node-local storage for Tier 1, the server pod cannot be rescheduled to another
    node while its node is down.

Problem

If the Klio Server pod or its node is unavailable for longer than the WAL headroom on
the primary allows (node failure, storage issue, Tier 1 PVC full, misconfiguration):

  1. The klio slot stops advancing.
  2. The primary retains all WAL since the slot's restart_lsn.
  3. The WAL PVC fills up and the primary stops accepting writes.

When synchronous mode is enabled, an unavailable Tier 1 could block commits even
earlier, unless quorum with other standbys is configured.

Today the only safeguards are on the user side:

  • max_slot_wal_keep_size, which deliberately breaks WAL continuity (the slot gets
    invalidated, leaving a gap in the archive and requiring a new base backup),
  • oversizing the WAL PVCs, auto-expanding them, and alerting.

Proposals (any of these would help)

  1. Multi-replica / active-passive Klio Server: e.g. a standby replica with leader
    election that can take over WAL streaming, with its own or replicated storage.
  2. Degraded mode / fallback: when Tier 1 is unreachable for a configurable time,
    let the sidecar ship WAL directly to Tier 2 (object storage), so the slot keeps
    advancing and WAL continuity is preserved.
  3. Built-in safeguard: an optional plugin-level threshold (retained WAL bytes or
    time) after which Klio takes a documented action, with clear events and metrics,
    instead of relying only on max_slot_wal_keep_size.
  4. Documentation of failure semantics, in particular:
    • behavior when the Klio Server is down for a long time,
    • how and when the klio slot is advanced ("controlled replication slot advancement"),
    • what happens to the klio slot on CNPG failover or switchover,
    • recovery procedure after the slot was invalidated (wal_status = 'lost'): does
      Klio detect it and trigger or require a new base backup?
    • guidance for running Tier 1 in multi-node / multi-AZ setups.
  5. Metrics and alerts: retained WAL behind the klio slot, streaming
    connected/disconnected state, and Tier 1 PVC usage (the docs note that there are
    no built-in storage alerts), plus example PrometheusRules.

Why this matters

Klio's design (WAL streaming, deduplication, tiering) is very attractive, but for
production adoption the backup path must not become a single point of failure for the
database itself. With archive_command-based solutions the same risk exists, but
operators are familiar with it. With Klio, Tier 1 adds an in-cluster component whose
availability directly affects the primary.

We're happy to test proposed solutions and share results.

主要语言
Go
星标
40
派生
10
平均合并
19 小时 58 分钟
30 天内合并 PR
175

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

cloudnative-pg/klio 的其他 Issue

查看 cloudnative-pg/klio 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。