borgbackup/borg

evaluate redundancy / error correction options

Open

#225 opened on Sep 30, 2015

View on GitHub
 (65 comments) (42 reactions) (0 assignees)Python (10,086 stars) (732 forks)batch import
help wantedquestion

Description

There is some danger that bitrot and storage media defects could lead to backup data loss / repository integrity issues. Deduplicating backup systems are more vulnerable to this than non-deduplicating ones, because a defect chunk affects all backup archives using this chunk.

Currently, there is a lot of error detection (CRCs, hashes, HMACs) going on in borgbackup, but it has no built-in support for error correction (see the FAQ about why), but it could be solved maybe using one of these approaches:

  • use borg to have N (N>1) independent backup repos of your data on different targets (if N-1 targets get corrupt then, you have still 1 working left. note that there is no support to create one non-corrupt repo from 2 corrupt repos, although that might be theoretically possible for some cases).
  • snapraid
  • par2
  • FECpp https://github.com/randombit/fecpp (BSD, C++ - make available via Cython?)
  • zfec (GPL/TGPPL, Python 2.x only, PR for Python 3.x exists)
  • RAID (and monitor and scrub the disks), ZFS mirror or RAIDZ* (better not use raid5 or raidz1)
  • zfs copies=N option (N>1)
  • specific filesystems
  • ceph librados
  • https://github.com/Bulat-Ziganshin/FastECC

If we can find some working approaches, we could add them to the documentation. Help and feedback about this is welcome!

Contributor guide

Tech stack
pythonshell
Domain
dataperformancedocumentation
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
understanding of backup systemsfamiliarity with error detection/correction concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
25
Research direction
Investigate and evaluate the listed error correction options (e.g., par2, FECpp, zfec, RAID, ZFS) for integration into borgbackup. Focus on feasibility, license compatibility, and performance impact. Document findings and propose documentation updates to the borgbackup repository, specifically in the FAQ or advanced usage sections. Consider contributing to existing PRs or discussions on Python 3 support for zfec.