Sync down improvements

Aperta
  • Qualcosa non va?
#722 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Da chiarire
Stato di attività
Ferma
Stack tecnologico
cloud, python
Ambito
cli, cloud

Direzione di ricerca

The sync-down download path and the --threads option are the starting points, with issue #720 as the referenced report of timeouts. The issue lists several separate proposals (per-file thread count, resumable downloads with a journal file, parallel parts, a .crdownload-style temp name) that are marked as backwards incompatible and needing a major version. Done would require a maintainer decision on which of these to pursue, so it is not a newcomer task yet.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Currently large file downloads use 10 threads per file, but --threads tells the sync process how many files to process at the same time, with a default of 10 --threads it's 100 download threads in total, which can cause some of them to time out if the source cluster has a high TTFB and local storage device is not fast enough. This has been observed in #720, but also others.

Configurability of download thread count should be improved.

Continuing broken downloads should be implemented somehow. As of today the native B2 server API does not allow for checking of checksums on the cloud side. Most likely it requires a separate "journal" file that would be flushed whenever we write a block, to indicate which parts of the file are written correctly (in case we use parallel downloader), or up to which point the file is correct.

B2 integration checklist says:

Downloads over 200MB should be split into parts and downloaded simultaneously. Once all parts are downloaded, the large file should be stitched together.
out of concern for performance, I think, but maybe if we have multiple download operations running simultaneously (due to sync down of many big files), then splitting them to smaller chunks is not good for anyone.

Finally, if a download operation is interrupted forcibly somehow, the client has a file with correct extension, maybe even correct size, but not necessarily correct data. In order to avoid this, Google Chrome adds a .crdownload extra extension to a file to indicate is has not been finished yet.

Most of those changes are backwards incompatible and would require a major version increase, so they should probably be done together.

Lingua principale
Python
Stelle
638
Fork
138
Merge medio
8h 27m
PR unite (30g)
8

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Backblaze/B2_Command_Line_Tool

Tutte le issue di Backblaze/B2_Command_Line_Tool

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.