Add adaptive --checkers=auto and make default

What is your current rclone version (output from rclone version)?

rclone v1.66.0-DEV

• os/version: debian 12.5 (64 bit)
• os/kernel: 6.1.0-14-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.21.8
• go/linking: dynamic
• go/tags: cmount

What problem are you are trying to solve?

When doing heavy I/O and local calculations (hashing) like rclone check or rclone cryptcheck, the default of --checkers=8 is often much too high, and ends up much slower than single threaded rclone check --checkers=1 on single spindle disk, in fact almost 3x slower due to thrashing the disk. When run on a dual-drive mirror, --checkers=2 is fastest. In my test cases, even with a 4-core/8-thread cpu, running with the default --checkers=8 also drove the load average up above 25 (due to disk contention, not cpu use), which would likely impact other processing (even though my tests were on an otherwise idle machine).

It seems like this would be much better to be somewhat adaptive, start with 1 or 2 checkers, and periodically see if adding/subtracting another checker increases or decreases total effective throughput.

How do you think rclone should be changed to solve that?

I propose that there should be a new auto (or maybe 0/-1?) setting for --checkers that is made the default, which will start at a much lower number of checkers (perhaps even 1) and then either add or subtract checkers as needed to adaptively optimize performance.

This would significantly help operations on non-SSD, non-raid/mirrored devices, and even on SSD's may improve performance over the default of 8 checkers due to SSD's tending to be much faster at contiguous reads than random reads. It may also help reduce system impact while improving operation speed.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.