sharkdp/fd

scatter search targets of threads to optimize search across multiple devices

Open

#559 opened on Apr 6, 2020

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Rust (42,984 stars) (1,059 forks)batch import
feature-requesthelp wantedperformance

Description

when searching across multiple devices, cpu utilization is rarely the bottleneck but rather io is.

an option to scatter the search targets of threads would speed up search massively when searching in /media/ for example across multiple external harddrives.

they may share a data link but its their latency and responsiveness that is the bottleneck. and can be exercised in parallel.

Contributor guide

Tech stack
rust
Domain
cliperformance
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
Rustasync I/O
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue proposes scattering search targets across threads to improve I/O parallelism when searching multiple drives. To implement, investigate the current threading model in the fd codebase, likely in src/worker.rs or similar. Consider using async I/O or a thread pool for I/O bound tasks. The discussion in the issue may provide additional context; look for comments suggesting design alternatives.