Periodic etcd Backup and Standalone Restore for Service Kubernetes

The payload names no existing files, so the first step is to find the current etcd backup task inside the K8s upgrade workflow and the rollback restore path. Read how they use etcdctl and etcdutl and where upgrade-status variables leak into them. The work splits into a backup playbook, a systemd timer, retention pruning and a separate restore playbook, so it needs a maintainer-agreed design before coding. Done means each scenario under Req 1 to Req 6 is covered and the upgrade and rollback paths are unchanged.

Periodic etcd Backup and Standalone Restore for Service Kubernetes

---

Summary

Infrastructure Administrators can schedule automated periodic etcd snapshots for the Service Kubernetes cluster, with configurable retention and snapshot verification, and independently restore etcd from any retained snapshot without invoking the full upgrade or rollback workflow. The feature re-uses and decouples the existing etcd backup logic from the upgrade workflow into a standalone Day-2 playbook, adds scheduling via systemd timers, snapshot integrity verification via etcdctl snapshot status, retention rotation, and a separate restore playbook that performs only the etcd data restoration steps without package downgrade or version-specific upgrade state dependencies.

---

Problem Statement

The Service Kubernetes cluster stores all cluster state -- workload definitions, service configurations, secrets, and orchestrator metadata (including Slurm-as-K8s-service state via Slinky) -- in etcd. Today, etcd backups are only taken as part of the K8s upgrade workflow. This creates three operational gaps:

1. No standalone invocation: The backup task requires the full upgrade context (upgrade-status state, version-keyed paths, status gates). An administrator cannot trigger a backup without initiating an upgrade.
2. No scheduling or retention: There is no periodic mechanism. Snapshots accumulate in version-keyed directories with no rotation or pruning. If an upgrade has not been run recently, there may be no backup at all.
3. No independent restore: The restore path is embedded in a multi-stage rollback workflow that includes package downgrade, VIP reconfiguration, and post-validation. There is no way to restore etcd from a snapshot without executing the full rollback.

For a large-scale cluster, an etcd data loss event without a recent verified backup means a full Service K8s cluster rebuild -- requiring significant engineering time plus cascading impact on all services running on the cluster (Slurm controller, telemetry pipeline, BSS).

---

Value Proposition

Customer Value

Infrastructure Administrators gain confidence that their Service Kubernetes cluster state is continuously protected by automated, verified backups with configurable retention. In the event of etcd data loss (hardware failure, corruption, or operator error), administrators can restore to a known-good state in under 30 minutes using a standalone restore playbook, without needing to rebuild the entire cluster or invoke the upgrade/rollback workflow.

---

User Personas

Infrastructure Administrator (Infra Admin)

Manages the HPC cluster post-deployment and is responsible for Day-2 operations including backup, recovery, and maintenance. Interacts with the feature by configuring backup schedules and retention policies, monitoring backup health, and executing restore operations when needed. Needs automated, reliable backups that do not require manual intervention or deep etcd expertise.

HPC Systems Engineer

Responsible for cluster-level service configuration and orchestration. Relies on etcd integrity for Slurm-as-K8s-service state, telemetry pipeline configuration, and BSS metadata. Needs assurance that a restore operation recovers all service state without requiring per-service reconfiguration.

---

Use Cases

Use Case 1: Scheduled Backup Configuration

Actor: Infrastructure Administrator (Infra Admin)
Context: During Day-2 cluster operations, the administrator configures automated periodic etcd backups to protect against data loss between upgrade cycles
Action: The administrator sets backup schedule (e.g., daily), retention count (e.g., keep last 7 snapshots), and NFS backup destination via configuration, then runs the backup setup playbook to deploy the systemd timer
Result: etcd snapshots are taken automatically at the configured interval, verified for integrity, stored on NFS with member-list metadata, and old snapshots beyond the retention count are pruned

Use Case 2: Standalone Disaster Recovery

Actor: Infrastructure Administrator (Infra Admin)
Context: An etcd data loss event occurs (disk failure, corruption, or accidental deletion) and the administrator needs to restore the Service K8s cluster to a known-good state without performing a full cluster rebuild
Action: The administrator runs the standalone restore playbook, specifying a snapshot from the retained backup set
Result: etcd is restored from the selected snapshot across all control plane nodes, kubelet and etcd services resume, and cluster workloads (Slurm controller, telemetry, BSS) recover to the state captured in the snapshot -- all within 30 minutes and without package downgrade or upgrade-state dependencies

Use Case 3: Ad-Hoc Pre-Maintenance Backup

Actor: HPC Systems Engineer
Context: Before performing a risky cluster operation (e.g., Slurm configuration change, CNI migration, or manual etcd compaction), the engineer wants to capture a point-in-time etcd snapshot as a safety net
Action: The engineer runs the standalone backup playbook manually to capture an immediate snapshot outside the scheduled cycle
Result: A verified etcd snapshot is saved to NFS alongside the scheduled backups, tagged with a user-provided label, and subject to the same retention policy

---

Scope Definition

Goals (Execution Outcomes)

• Goal 1: Provide a standalone Day-2 backup playbook that captures an etcd snapshot, copies it to NFS, verifies snapshot integrity, and records member-list metadata -- without requiring upgrade context or upgrade-status state.
• Goal 2: Provide scheduling via systemd timer deployment, with configurable interval (default: daily) and retention count (default: 7 snapshots).
• Goal 3: Provide a standalone restore playbook that restores etcd from a specified snapshot across all control plane nodes, restarts kubelet, and verifies cluster health -- without invoking package downgrade, BSS restore, or other rollback stages.
• Goal 4: Re-use the existing etcd backup and restore task logic where possible, decoupling from upgrade-specific variables and status tracking.
• Goal 5: Support both NFS-backed etcd (default) and local-disk etcd configurations.

Non-Goals (Scope Boundaries)

• Non-Goal 1: Modifying the existing upgrade or rollback workflows -- those paths remain unchanged and continue to use their own backup/restore logic.
• Non-Goal 2: Providing a Kubernetes-native backup operator (e.g., Velero) -- this feature uses Ansible playbooks consistent with the existing operational model.
• Non-Goal 3: Backing up Kubernetes application-level state (PVCs, ConfigMaps, Secrets beyond etcd) -- this feature captures the etcd datastore only; application-level DR is out of scope.
• Non-Goal 4: Encrypting snapshots at rest -- snapshots are stored as plain .db files on NFS consistent with existing conventions; encryption-at-rest is a separate concern.
• Non-Goal 5: Multi-cluster support -- this feature targets the primary service K8s cluster consistent with the current single-cluster architecture.

---

Requirements

Requirement: Standalone etcd Backup [Req 1]

The system must provide a standalone playbook that captures an etcd snapshot on the active control plane node, copies the snapshot and etcdctl binary to NFS, captures the etcd member list as JSON, and verifies snapshot integrity -- all without requiring upgrade-status state or version-keyed paths.

Relates to: UC-1 -- Scheduled Backup Configuration, UC-3 -- Ad-Hoc Pre-Maintenance Backup

Scenario: Successful standalone backup

• When the administrator runs the standalone backup playbook, the system takes an etcd snapshot using TLS certificates, copies the snapshot to a timestamped directory on NFS, captures the member list, verifies the snapshot via etcdctl snapshot status, and logs success.

Scenario: etcd endpoint unreachable

• When the administrator runs the standalone backup playbook and the etcd endpoint is unreachable, the system retries with configurable delay, then fails with a clear error message identifying the connectivity issue.

---

Requirement: Snapshot Integrity Verification [Req 2]

The system must verify the integrity of every etcd snapshot immediately after creation by running etcdctl snapshot status and recording the revision, hash, total keys, and total size.

Relates to: UC-1 -- Scheduled Backup Configuration

Scenario: Snapshot verification passes

• When the snapshot is saved to NFS, the system runs etcdctl snapshot status, records the output alongside the snapshot, and logs that verification passed.

Scenario: Snapshot verification fails

• When etcdctl snapshot status reports an error or inconsistent hash, the system marks the snapshot as failed, does not count it toward retention, and logs a high-severity warning.

---

Requirement: Configurable Scheduling [Req 3]

The system must deploy a systemd timer on the active control plane node to execute the backup playbook at a configurable interval with a configurable retention count.

Relates to: UC-1 -- Scheduled Backup Configuration

Scenario: Timer deployment with defaults

• When the administrator runs the backup setup playbook without overriding defaults, the system deploys a systemd timer that runs daily and retains the last 7 verified snapshots.

Scenario: Custom schedule and retention

• When the administrator configures a custom schedule and retention count, the system deploys a timer matching the specified interval and retains the specified number of verified snapshots.

---

Requirement: Retention and Rotation [Req 4]

The system must prune snapshots that exceed the configured retention count, keeping only the most recent verified snapshots and their associated metadata files.

Relates to: UC-1 -- Scheduled Backup Configuration

Scenario: Pruning after retention count exceeded

• When the number of verified snapshots on NFS exceeds the retention count, the system deletes the oldest snapshots and their metadata until the count matches the configured retention.

Scenario: Failed snapshots excluded from retention count

• When a snapshot fails verification, the system does not count it toward the retention limit and deletes it during the next pruning cycle.

---

Requirement: Standalone etcd Restore [Req 5]

The system must provide a standalone restore playbook that restores etcd from a specified snapshot across all control plane nodes without invoking package downgrade, BSS restore, VIP reconfiguration, or other upgrade/rollback stages.

Relates to: UC-2 -- Standalone Disaster Recovery

Scenario: Successful restore from snapshot

• When the administrator runs the standalone restore playbook specifying a snapshot path, the system stops etcd on all control plane nodes, restores the snapshot using etcdutl with the correct initial-cluster configuration from the member-list JSON, restarts kubelet on all control plane nodes, and verifies etcd cluster health.

Scenario: Snapshot file not found

• When the administrator specifies a snapshot path that does not exist, the system fails immediately with a clear error message listing available snapshots in the backup directory.

Scenario: Partial restore failure

• When the restore succeeds on some control plane nodes but fails on others, the system logs which nodes failed, does not restart kubelet on failed nodes, and provides clear remediation guidance.

---

Requirement: Backup Catalog and Discoverability [Req 6]

The system must maintain a machine-readable backup catalog on NFS that lists all retained snapshots with their timestamp, verification status, etcd revision, and size.

Relates to: UC-2 -- Standalone Disaster Recovery

Scenario: Catalog updated after backup

• When a backup completes (success or failure), the system appends an entry to the catalog file with the snapshot path, timestamp, verification result, revision number, and size in bytes.

Scenario: Administrator lists available backups

• When the administrator queries the catalog, the system displays all retained snapshots sorted by timestamp with their verification status.

---

Reliability

Failure Mode	System Behavior
etcd endpoint unreachable during backup	Retry with configurable delay; fail with clear error if all retries exhausted
NFS mount unavailable during backup	Fail immediately with error identifying the NFS mount point
Snapshot verification failure	Mark snapshot as failed; do not count toward retention; log high-severity warning
Restore fails on one or more control plane nodes	Log per-node status; do not restart kubelet on failed nodes; provide remediation steps
systemd timer fails to fire	Timer logs to journald; administrator can check timer status via standard systemctl commands

Availability: Backup timer runs on the active control plane node; backup availability is bounded by control plane and NFS availability.
Recovery: Failed backups do not affect cluster state. Failed restores leave the cluster in the pre-restore state (etcd data dir is backed up before overwrite).

---

Security

• Backup and restore playbooks execute using the Infrastructure Administrator credentials via Ansible, consistent with the existing operational model.
• etcd TLS certificates are used for all etcdctl operations. No additional certificates or credentials are required.
• Snapshots contain the full etcd datastore including Kubernetes Secrets -- access to the NFS backup directory must be restricted to Infrastructure Administrators.

---

Telemetry

• Engineering should expose backup success/failure counts as a Prometheus metric to the central observability stack, enabling alerting on missed or failed backups.
• Each backup should record: snapshot size (bytes), etcd revision at snapshot time, snapshot creation duration, and verification result.

---

Performance

Metric	Target
Snapshot creation time (single control plane node)	< 60 seconds
Snapshot verification time	< 10 seconds
Snapshot copy to NFS	< 120 seconds
Full restore (3-node control plane)	< 30 minutes
Retention pruning	< 30 seconds

---

APIs

Interface	Type	Consumer	Details
etcd_backup_schedule	Configuration	Ansible / Administrator	Systemd timer OnCalendar expression (default: daily at 2 AM)
etcd_backup_retention_count	Configuration	Ansible / Administrator	Number of verified snapshots to retain (default: 7)
etcd_backup_nfs_path	Configuration	Ansible / Administrator	NFS directory for snapshots (default: derived from storage configuration)
etcd_backup_catalog	Data File	Restore playbook / Administrator	JSON-lines file on NFS listing all retained snapshots with metadata

---

Documentation

• Operational Runbook for Infrastructure Administrators covering: backup configuration (schedule, retention), manual ad-hoc backup execution, restore procedure with step-by-step instructions, backup monitoring and troubleshooting, and catalog inspection.
• Developer Documentation covering: architecture of the standalone backup/restore playbooks, relationship to existing upgrade/rollback backup code, systemd timer configuration, and catalog format specification.
• Integration Notes for HPC Systems Engineers documenting which service state is captured in etcd and expected behavior after a restore.

---

Scope & Sequencing

N/A -- This is a single-phase feature. All capabilities (standalone backup, verification, scheduling, retention, standalone restore, and catalog) will be delivered in the initial implementation.

---

Dependencies

• No upstream spec dependencies. This feature is standalone and does not block or depend on any other backlog item.

Internal Dependencies

Existing etcd backup and restore task logic from upgrade/rollback workflows, NFS storage configuration, and high availability configuration for control plane node resolution.

Platform Dependencies

• Kubernetes: Service K8s cluster with kubeadm-managed etcd (etcd 3.6.x)
• Operating System: RHEL 10.0 and later
• etcd utilities: etcdctl (for backup and health checks) and etcdutl (for restore)
• Storage: NFS mount available from control plane nodes
• systemd: For timer-based scheduling on control plane nodes

---

Risks

Risk	Category	Impact	Likelihood	Mitigation
Snapshot taken during etcd compaction or heavy write load may be inconsistent	Technical	Medium	Low	Snapshot verification catches inconsistent snapshots; failed snapshots are excluded from retention and trigger alerts
NFS storage fills up due to large or frequent snapshots	Operational	Medium	Medium	Retention pruning limits snapshot count; backup logs snapshot size; documentation recommends monitoring NFS capacity
Restore disrupts running workloads	Operational	High	Low	Restore playbook requires explicit confirmation; documentation warns that all changes since the snapshot will be lost; recommend draining workloads before restore
systemd timer not firing due to node reboot or kubelet restart	Technical	Low	Medium	Timer is enabled and started via systemctl; journald logging allows health monitoring; missed backup windows are detected by catalog gap analysis

---

Acceptance Criteria

• AC-001: The standalone backup playbook captures an etcd snapshot, copies it to a timestamped directory on NFS, captures the member list as JSON, and verifies the snapshot -- all without requiring upgrade-status state or version-keyed paths [Req 1]
• AC-002: The standalone backup playbook retries when the etcd endpoint is unreachable, then fails with a clear error [Req 1]
• AC-003: Every snapshot is verified immediately after creation, and the verification output (revision, hash, keys, size) is recorded alongside the snapshot [Req 2]
• AC-004: A snapshot that fails verification is marked as failed, not counted toward retention, and triggers a high-severity log warning [Req 2]
• AC-005: The backup setup playbook deploys a systemd timer with the configured schedule and retention count [Req 3]
• AC-006: Snapshots exceeding the retention count are pruned, keeping only the most recent verified snapshots and their metadata [Req 4]
• AC-007: The standalone restore playbook restores etcd from a specified snapshot across all control plane nodes, restarts kubelet, and verifies cluster health -- without invoking package downgrade, BSS restore, or other rollback stages [Req 5]
• AC-008: The restore playbook fails with a clear error listing available snapshots when the specified snapshot path does not exist [Req 5]
• AC-009: A machine-readable backup catalog on NFS lists all retained snapshots with timestamp, verification status, revision, and size [Req 6]

---

Assumptions / Open Questions

Assumptions

• A1: Required underlying hardware and OS capabilities are present.

Open Questions

• None.

---