borgbackup/borg

ideas for using the flags

Open

#8514 opened on Nov 1, 2024

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (10,086 stars) (732 forks)batch import
c: cachec: hashindexhelp wanted

Description

The ChunkIndex is a mapping 256bit key --> (32bit flags, 32bit size) since #8513.

The user flags are:

  • F_USED, meaning the chunk is referenced / used. this is used by borg compact to determine which chunks are used / which are not used (and then deleting the unused chunks from the repo).
  • F_COMPRESS, meaning the chunk shall get (re-)compressed, used by borg repo-compress.

The system flags are:

  • F_NEW, meaning the chunk was added after the last saving of the chunk index (flagging the chunks that need to be saved next)

Other flag bits are not used yet and available for creative usage.

Besides using these bits as flags/markers of some kind, we can also do memory-efficient (no additional memory use!) set operations, like e.g. to compute a set intersection:

  • set bit1 for members of set1
  • set bit2 for members of set2
  • intersection set1 & set2: iterate over all entries in the ChunkIndex, entries with both bits set are members of the intersection.

Collect any ideas what can be done with that below.

Contributor guide

Tech stack
python
Domain
dataperformance
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Pythonborg internalschunk index concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40
Research direction
This issue is a brainstorming request for creative uses of the 32 bit flags in Borg's ChunkIndex. Currently, only F USED, F COMPRESS, and F NEW are used, with many bits free. The goal is to propose new uses for these flags, including memory efficient set operations via bitwise logic. To contribute, read the linked PR (#8513) and comments to understand the current flag semantics, then post your ideas in the thread. No code changes are required, only thoughtful suggestions.