ipfs/kubo

Tune batch add parallelism constant

Open

#4299 opened on Oct 11, 2017

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (13,906 stars) (2,725 forks)batch import
help wantedstatus/deferredtopic/perf

Description

Specifically, tune the ParallelBatchCommits constant added in #4296. Right now, we just use NumCPU*2 but we could probably do something fancier (learn it? does the datastore matter? does the storage medium matter? how about the filesystem?).

We may never get to this but hey, if anyone wants to spend some time experimenting they're free to take a crack at it.

Contributor guide

Tech stack
go
Domain
performance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
unclear
Prerequisites
GoIPFS conceptsperformance profilingunderstanding of batch commit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
Investigate the `ParallelBatchCommits` constant added in #4296. Experiment with different values and measure performance across various datastores, storage media, and filesystems. Consider implementing adaptive tuning based on runtime metrics. Refer to the codebase for how the constant is used in batch add operations.