ipfs/kubo

enhancement: tarsnap chunker / importer (Content defined, polynomial, fast converging chunking)

Open

#3,603 opened on Jan 17, 2017

View on GitHub
 (1 comment) (2 reactions) (0 assignees)Go (2,725 forks)batch import
help wantedkind/enhancement

Repository metrics

Stars
 (13,906 stars)
PR merge metrics
 (Avg merge 4d 2h) (28 merged PRs in 30d)

Description

Version information:

go-ipfs version: 0.4.4

Type: Feature, Enhancement

Priority: P4

Area: Tools, Importer

Description:

Better suited for maximizing deduplication ratio then current Rabin chunker.
Using smaller chunks with faster convergence yields greater space savings, and the benefit depending on dataset can be great in comparison to Rabin.

The mean chunk size used by tarsnap is 64k.

Source: https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.h https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.c

Related: https://moinakg.wordpress.com/2012/11/11/inside-content-defined-chunking-in-pcompress/ https://moinakg.wordpress.com/2012/11/15/inside-content-defined-chunking-in-pcompress-part-2/

Contributor guide

Research direction
Study the existing Rabin chunker implementation in the codebase, then design and implement a new content defined chunker based on the tarsnap references provided. Focus on the polynomial hashing and fast convergence techniques.
Tech stack
go
Domain
backendperformance
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20