High availability for the Klio Server (Tier 1): avoid WAL accumulation on the primary when Tier 1 is unavailable

Aperta
  • Qualcosa non va?
#513 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
go, kubernetes, postgresql
Ambito
databases, devops, infrastructure

Direzione di ricerca

Start by reading the documentation sections on Managing Storage and the Server custom resource, then trace how the klio physical replication slot is created and advanced. The issue proposes several different directions, from HA to safeguards, documentation, and metrics, so agree on a specific scope before starting; done means implementing that scope and documenting the behavior and recovery steps.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

The Klio Server (Tier 1) currently runs as a single-replica StatefulSet backed by a PVC.
Because WAL is streamed through the physical replication slot klio, any prolonged
unavailability of Tier 1 makes PostgreSQL retain WAL on the primary indefinitely.
A backup-infrastructure outage can therefore turn into a database outage once the
primary's WAL volume fills up.

We'd like to request an HA story for Tier 1, or at least documented failure semantics
and built-in safeguards.

Context

  • Klio version: 0.0.20
  • CloudNativePG on RKE2, PostgreSQL 18
  • PostgreSQL data/WAL on node-local storage (TopoLVM), which is a common setup for CNPG
  • Two data centers, no stretched storage

As far as we can tell from the documentation:

  • The Server CR creates a StatefulSet with a single server container
    (pod <name>-klio-0); replicas, failover and leader election are not mentioned.
  • The klio-plugin sidecar creates the physical slot klio on the primary.
  • "Managing Storage" states that when Tier 1 is full, WAL streaming stops and
    "PostgreSQL accumulates WAL on its own PVC".
  • With node-local storage for Tier 1, the server pod cannot be rescheduled to another
    node while its node is down.

Problem

If the Klio Server pod or its node is unavailable for longer than the WAL headroom on
the primary allows (node failure, storage issue, Tier 1 PVC full, misconfiguration):

  1. The klio slot stops advancing.
  2. The primary retains all WAL since the slot's restart_lsn.
  3. The WAL PVC fills up and the primary stops accepting writes.

When synchronous mode is enabled, an unavailable Tier 1 could block commits even
earlier, unless quorum with other standbys is configured.

Today the only safeguards are on the user side:

  • max_slot_wal_keep_size, which deliberately breaks WAL continuity (the slot gets
    invalidated, leaving a gap in the archive and requiring a new base backup),
  • oversizing the WAL PVCs, auto-expanding them, and alerting.

Proposals (any of these would help)

  1. Multi-replica / active-passive Klio Server: e.g. a standby replica with leader
    election that can take over WAL streaming, with its own or replicated storage.
  2. Degraded mode / fallback: when Tier 1 is unreachable for a configurable time,
    let the sidecar ship WAL directly to Tier 2 (object storage), so the slot keeps
    advancing and WAL continuity is preserved.
  3. Built-in safeguard: an optional plugin-level threshold (retained WAL bytes or
    time) after which Klio takes a documented action, with clear events and metrics,
    instead of relying only on max_slot_wal_keep_size.
  4. Documentation of failure semantics, in particular:
    • behavior when the Klio Server is down for a long time,
    • how and when the klio slot is advanced ("controlled replication slot advancement"),
    • what happens to the klio slot on CNPG failover or switchover,
    • recovery procedure after the slot was invalidated (wal_status = 'lost'): does
      Klio detect it and trigger or require a new base backup?
    • guidance for running Tier 1 in multi-node / multi-AZ setups.
  5. Metrics and alerts: retained WAL behind the klio slot, streaming
    connected/disconnected state, and Tier 1 PVC usage (the docs note that there are
    no built-in storage alerts), plus example PrometheusRules.

Why this matters

Klio's design (WAL streaming, deduplication, tiering) is very attractive, but for
production adoption the backup path must not become a single point of failure for the
database itself. With archive_command-based solutions the same risk exists, but
operators are familiar with it. With Klio, Tier 1 adds an in-cluster component whose
availability directly affects the primary.

We're happy to test proposed solutions and share results.

Lingua principale
Go
Stelle
40
Fork
10
Merge medio
19h 58m
PR unite (30g)
175

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di cloudnative-pg/klio

Tutte le issue di cloudnative-pg/klio

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.