High availability for the Klio Server (Tier 1): avoid WAL accumulation on the primary when Tier 1 is unavailable

Abierto
  • ¿Algo no está bien?
#513 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, kubernetes, postgresql
Área
databases, devops, infrastructure

Línea de trabajo

Start by reading the documentation sections on Managing Storage and the Server custom resource, then trace how the klio physical replication slot is created and advanced. The issue proposes several different directions, from HA to safeguards, documentation, and metrics, so agree on a specific scope before starting; done means implementing that scope and documenting the behavior and recovery steps.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

The Klio Server (Tier 1) currently runs as a single-replica StatefulSet backed by a PVC.
Because WAL is streamed through the physical replication slot klio, any prolonged
unavailability of Tier 1 makes PostgreSQL retain WAL on the primary indefinitely.
A backup-infrastructure outage can therefore turn into a database outage once the
primary's WAL volume fills up.

We'd like to request an HA story for Tier 1, or at least documented failure semantics
and built-in safeguards.

Context

  • Klio version: 0.0.20
  • CloudNativePG on RKE2, PostgreSQL 18
  • PostgreSQL data/WAL on node-local storage (TopoLVM), which is a common setup for CNPG
  • Two data centers, no stretched storage

As far as we can tell from the documentation:

  • The Server CR creates a StatefulSet with a single server container
    (pod <name>-klio-0); replicas, failover and leader election are not mentioned.
  • The klio-plugin sidecar creates the physical slot klio on the primary.
  • "Managing Storage" states that when Tier 1 is full, WAL streaming stops and
    "PostgreSQL accumulates WAL on its own PVC".
  • With node-local storage for Tier 1, the server pod cannot be rescheduled to another
    node while its node is down.

Problem

If the Klio Server pod or its node is unavailable for longer than the WAL headroom on
the primary allows (node failure, storage issue, Tier 1 PVC full, misconfiguration):

  1. The klio slot stops advancing.
  2. The primary retains all WAL since the slot's restart_lsn.
  3. The WAL PVC fills up and the primary stops accepting writes.

When synchronous mode is enabled, an unavailable Tier 1 could block commits even
earlier, unless quorum with other standbys is configured.

Today the only safeguards are on the user side:

  • max_slot_wal_keep_size, which deliberately breaks WAL continuity (the slot gets
    invalidated, leaving a gap in the archive and requiring a new base backup),
  • oversizing the WAL PVCs, auto-expanding them, and alerting.

Proposals (any of these would help)

  1. Multi-replica / active-passive Klio Server: e.g. a standby replica with leader
    election that can take over WAL streaming, with its own or replicated storage.
  2. Degraded mode / fallback: when Tier 1 is unreachable for a configurable time,
    let the sidecar ship WAL directly to Tier 2 (object storage), so the slot keeps
    advancing and WAL continuity is preserved.
  3. Built-in safeguard: an optional plugin-level threshold (retained WAL bytes or
    time) after which Klio takes a documented action, with clear events and metrics,
    instead of relying only on max_slot_wal_keep_size.
  4. Documentation of failure semantics, in particular:
    • behavior when the Klio Server is down for a long time,
    • how and when the klio slot is advanced ("controlled replication slot advancement"),
    • what happens to the klio slot on CNPG failover or switchover,
    • recovery procedure after the slot was invalidated (wal_status = 'lost'): does
      Klio detect it and trigger or require a new base backup?
    • guidance for running Tier 1 in multi-node / multi-AZ setups.
  5. Metrics and alerts: retained WAL behind the klio slot, streaming
    connected/disconnected state, and Tier 1 PVC usage (the docs note that there are
    no built-in storage alerts), plus example PrometheusRules.

Why this matters

Klio's design (WAL streaming, deduplication, tiering) is very attractive, but for
production adoption the backup path must not become a single point of failure for the
database itself. With archive_command-based solutions the same risk exists, but
operators are familiar with it. With Klio, Tier 1 adds an in-cluster component whose
availability directly affects the primary.

We're happy to test proposed solutions and share results.

Lenguaje dominante
Go
Estrellas
40
Forks
10
Merge medio
19 h 58 min
PR fusionados (30 d)
175

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de cloudnative-pg/klio

Todos los issues de cloudnative-pg/klio

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.