Add remote based on erasure coding

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/can-we-add-erasure-coding-to-rclone/23684/5

What is your current rclone version (output from rclone version)?

rclone v1.55.0

• os/type: linux
• os/arch: amd64
• go/version: go1.16.2
• go/linking: static
• go/tags: cmount

What problems are you are trying to solve?

1. In the past 1-2 weeks we have had major outages with both OneDrive and GoogleDrive. Nothing was lost but during the outages my files stored on the cloud provider were not accessible.
2. Rumors have it that the security around some of the cloud providers is not top-notch. I have read quite a few stories about for example OneDrive being easy to hack.
3. I do not like the idea that a cloud provider has a full copy of my data which they can peruse and use to their hearts content.
4. Sometimes for reasons unknown to me a cloud provider really slows down. Making it hard to access my files in a timely matter.

Problem 1 could be solved by using the union remote: that writes each data block to multiple upstream cloud providers. Problems with the union remote: however:

1. it doubles or triples or quadruples, ... the amount of storage it would use on the cloud providers.
2. it doubles or triples or quadruples, ... the amount of data that would need to be uploaded to the cloud providers.
3. if one of the upstreams is unavailable (or gives errors) the read will fail.

Problems 2 and 3 can be solved by always using an encrypted remote

Problem 4 can not really be solved with the current version of rclone (AFAIK)

How do you think rclone should be changed to solve that?

My proposal is to add a new remote: that uses erasure encoding.

For this remote type we specify for example 3(let's call it M) of 5(let's call it N) meaning we (erasure) encode the data block to be written to the remote: such that it will be striped over 5 clouds but we only need 3 clouds to be able to recreate (read) the original data. This will add reliability (solve problem 1) as well as "encryption" (solve problems 2 and 3) since the original data can only be reconstructed if 3 of the 5 clouds "cooperate" (or are hacked). It will also increase the download speed (solve problem 4) since when reading we send requests to all 5 clouds and use the first 3 answers we get to recreate the data. Which gives us the fastest possible response (and we are reading the data using parallel downloads).

Design:

rclone config
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Erasure encodes the contents across several remotes
   \ "erasure"
[snip]
Storage> erasure
List of space separated remotes.
Can be 'clouda:test/dir upstreamb:', '\"clouda:test/space:ro dir\" cloudb:', etc.
Enter a string value.
upstreams> remote1:dir1 remote2:dir2 remote3:dir3 remote4:dir4 remote5:dir5
Allowed number of inaccessible hosts. Enter an integer between 1-4  Press Enter for the default ("2").
maxhostsdown>

Remote config
--------------------
[remote]
type = erasure
upstreams = remote1:dir1 remote2:dir2 remote3:dir3 remote4:dir4 remote5:dir5
maxhostsdown = 2
overhead = XXXX%
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

The XXXX% is the percentage of overhead. It will be in the config file to inform the user of the amount of overhead. It will not be used for any configuration.

When writing to this remote: it should fail if not all upstreams are written correctly.

PS: I am free to discuss further design issues.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.