Sync down improvements
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
The sync-down download path and the --threads option are the starting points, with issue #720 as the referenced report of timeouts. The issue lists several separate proposals (per-file thread count, resumable downloads with a journal file, parallel parts, a .crdownload-style temp name) that are marked as backwards incompatible and needing a major version. Done would require a maintainer decision on which of these to pursue, so it is not a newcomer task yet.
由索引模型根据 Issue 内容生成。
描述
Currently large file downloads use 10 threads per file, but --threads tells the sync process how many files to process at the same time, with a default of 10 --threads it's 100 download threads in total, which can cause some of them to time out if the source cluster has a high TTFB and local storage device is not fast enough. This has been observed in #720, but also others.
Configurability of download thread count should be improved.
Continuing broken downloads should be implemented somehow. As of today the native B2 server API does not allow for checking of checksums on the cloud side. Most likely it requires a separate "journal" file that would be flushed whenever we write a block, to indicate which parts of the file are written correctly (in case we use parallel downloader), or up to which point the file is correct.
B2 integration checklist says:
Downloads over 200MB should be split into parts and downloaded simultaneously. Once all parts are downloaded, the large file should be stitched together.
out of concern for performance, I think, but maybe if we have multiple download operations running simultaneously (due to sync down of many big files), then splitting them to smaller chunks is not good for anyone.
Finally, if a download operation is interrupted forcibly somehow, the client has a file with correct extension, maybe even correct size, but not necessarily correct data. In order to avoid this, Google Chrome adds a .crdownload extra extension to a file to indicate is has not been finished yet.
Most of those changes are backwards incompatible and would require a major version increase, so they should probably be done together.
- 主要语言
- Python
- 星标
- 638
- 派生
- 138
- 平均合并
- 8 小时 27 分钟
- 30 天内合并 PR
- 8
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
Backblaze/B2_Command_Line_Tool 的其他 Issue
-
enhancement
难度 2/5 1-3 小时 新手友好度 62/100
Backblaze/B2_Command_Line_Tool#553 · 2 条评论 · 3 个 reaction ·
维护者通常 1 天内回复
-
Adopt the AI-native SDLC playbook and repository policies可能已有人在做 @goanpeca 于 18 天前认领。 未关闭enhancement
难度 5/5 一周以上 新手友好度 25/100
Backblaze/B2_Command_Line_Tool#1192 ·
维护者通常 1 天内回复
-
难度 5/5 一周以上 新手友好度 25/100
Backblaze/B2_Command_Line_Tool#1168 ·
维护者通常 1 天内回复
-
Add to MacPorts未关闭
难度 3/5 1-2 天 新手友好度 50/100
Backblaze/B2_Command_Line_Tool#1161 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 56/100
Backblaze/B2_Command_Line_Tool#1155 ·
维护者通常 1 天内回复
查看 Backblaze/B2_Command_Line_Tool 的全部 Issue
相似的 Issue
-
ライブラリを追加未关闭priority middle
难度 1/5 1 小时以内 新手友好度 62/100
KATO-Hiro/AtCoderClans#13015 ·
维护者通常 1 天内回复
-
StringLiteral to IntegerType/LongType silently truncates fractional strings, changing predicate meaning可能已有人在做 @Shahzeb-DataEngineer 今天认领。 未关闭kind:bug
难度 2/5 1-3 小时 新手友好度 68/100
apache/iceberg-python#4382 · 2 条评论 ·
维护者通常 1 天内回复
-
request-theme
难度 2/5 1 小时以内 新手友好度 62/100
LizardByte/ThemerrDB#10024 · 2 条评论 ·
-
难度 1/5 1 小时以内 新手友好度 72/100
btclib-org/btclib-node#2505 · 1 条评论 ·
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 78/100
open-webui/open-webui#34462 · 1 条评论 ·
维护者通常 1 天内回复