Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

feat: add backup/replication for the object-storage module's own bucket contents

Aperta
  • Qualcosa non va?
#669 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Ambito
cloud, data-engineering, infrastructure

Direzione di ricerca

Start by reading #663 to understand the proposed object-storage module and its provider, then compare its capabilities with the scheduled sync options described here and the file/volume backup path in #668. The work depends on #663 landing; done means bucket contents have a recovery path independent of that file/volume backup, with accidental deletion or overwrite recoverable.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement
Problem statement

#668 covers backing up file/volume-based module data (Dagster IO manager, DuckDB, dlt) using restic, with S3 as one possible destination. It does not cover the case where the object-storage module proposed in #663 is itself added to a stack -- at that point, the bucket's own contents become another persistence layer (uploaded files, dbt artifacts, Superset uploads, IO manager objects, etc.) that can be lost to accidental deletion, corruption, or an operator mistake, with no protection of its own.

Proposed solution

Once #663 lands, add a way to protect the object-storage module's bucket contents independently of the file/volume backup path in #668 -- e.g. scheduled cross-bucket replication/sync (to a second bucket, ideally a different backend/host) or bucket versioning where the chosen provider supports it, so an accidental delete/overwrite is recoverable.

Leading open-source candidates: rclone (MIT) or mc mirror (the MinIO client CLI, Apache 2.0, still maintained even though the MinIO server itself is EOL per #663's findings) for scheduled bucket-to-bucket sync; enabling native object versioning on the storage module if the selected provider (e.g. Garage, SeaweedFS) supports it, as a lower-effort first line of defense.

Alternatives considered
1. Rely on restic (#668) against bucket contents directly

Possible in principle (restic can back up any mounted filesystem), but object-storage backends aren't normally exposed as a filesystem to the backup process, and restic is not designed for S3-to-S3 semantics (versioning, multipart objects) -- a native replication/versioning approach is a better fit than forcing this through #668's file/volume path.

2. No bucket-content protection (status quo once #663 ships)

Leaves the object-storage module as the one persistence layer in the stack with no recovery path at all, even after #665 (Postgres) and #668 (files/volumes) are addressed.

Impact
  • Depends on #663 (object-storage module) existing first; not actionable before that lands.
  • Complements #665 (Postgres) and #668 (file/volume backup) so all three of CDS's persistence patterns (SQL database, file/volume, object storage) have a recovery story.
Lingua principale
Python
Stelle
8
Fork
12
Merge medio
1g 2h
PR unite (30g)
93

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di RonaldHensbergen/composable-data-stack

Tutte le issue di RonaldHensbergen/composable-data-stack

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.