Repository metrics
- Stars
- (0 stars)
- PR merge metrics
- (PR metrics pending)
Description
Currently when making a snapshot we use a hash accumulator to determine if the contents are identical on the source and target. This is great but this does not help with a very large database to determine where the failure exists.
We therefore should create a merklized snapshot that compresses N records per leaf. Let's say every 10K records we add a leaf that is the hash accumulator valut to a merkle tree, grow this tree to size M; once full store the merkle root in a new merkle tree that records snpshots of snapshots.
With this we can create a skip list to determine much quicker where failures are at.
The merkle is just an idea, we can also keep a simple list of hashes per N records, point is, we need to be able to find data coherency issues and reconcile them without having to traverse all records.