ipfs/kubo

enhancement: tarsnap chunker / importer (Content defined, polynomial, fast converging chunking)

Open

#3603 opened on Jan 17, 2017

View on GitHub
 (1 comment) (2 reactions) (0 assignees)Go (13,906 stars) (2,725 forks)batch import
help wantedkind/enhancement

Description

Version information:

go-ipfs version: 0.4.4

Type: Feature, Enhancement

Priority: P4

Area: Tools, Importer

Description:

Better suited for maximizing deduplication ratio then current Rabin chunker.
Using smaller chunks with faster convergence yields greater space savings, and the benefit depending on dataset can be great in comparison to Rabin.

The mean chunk size used by tarsnap is 64k.

Source: https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.h https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.c

Related: https://moinakg.wordpress.com/2012/11/11/inside-content-defined-chunking-in-pcompress/ https://moinakg.wordpress.com/2012/11/15/inside-content-defined-chunking-in-pcompress-part-2/

Contributor guide

Tech stack
go
Domain
data
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Go programmingIPFS basicsContent defined chunking
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Study the tarsnap chunker algorithm from the provided source files (chunkify.h and chunkify.c) and the two referenced blog posts. Then examine the current IPFS importer code in the repository to understand how to integrate a new chunker type. Since there is no prior discussion, consider asking the maintainers for guidance on the expected interface and testing approach.