ipfs/kubo

Tune batch add parallelism constant

Open

#4,299 opened on Oct 11, 2017

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (2,725 forks)batch import
help wantedstatus/deferredtopic/perf

Repository metrics

Stars
 (13,906 stars)
PR merge metrics
 (Avg merge 4d 2h) (28 merged PRs in 30d)

Description

Specifically, tune the ParallelBatchCommits constant added in #4296. Right now, we just use NumCPU*2 but we could probably do something fancier (learn it? does the datastore matter? does the storage medium matter? how about the filesystem?).

We may never get to this but hey, if anyone wants to spend some time experimenting they're free to take a crack at it.

Contributor guide

Research direction
Set up benchmarks to measure commit throughput under varying ParallelBatchCommits values. Analyze impact of different datastores and storage media. Consider adaptive or learned approaches.
Tech stack
go
Domain
backendperformance
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
Go
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30