ipld/go-car

blockstore: OpenReadWrite should use atomic writes

開放

#249 建立於 2021年9月30日

 (1 則留言) (0 個反應) (0 位負責人)Go (54 個分叉)github user discovery
P3help wanted

倉庫指標

星標
 (176 顆星)
PR 合併指標
 (平均合併 23小時 30分鐘) (30 天內合併 12 個 PR)

描述

Right now, OpenReadWrite writes directly to the destination file. This has multiple problems:

  • If we encounter some error mid-finalize, we may leave a corrupted file
  • If we encounter a write/flush error while finalizing, we may leave a partial file
  • If we're resuming on an existing file and writing in-place, we may corrupt the user's data (e.g. https://github.com/ipld/go-car/issues/247)

It would be much saner overall to instead write to a temporary copy next to the destination file (such as foo.car.tmp for foo.car), and once Finalize has finished with no error, we do a rename that should basically never fail.

This also means that we can teach OpenReadWrite to always remove foo.car.tmp when it's discarded, to ensure we don't leave unfinished business behind.

The method described above should be doable with just a bit of os and io glue. We could use https://pkg.go.dev/github.com/google/renameio#TempFile, though I think pulling in a library for this use case is perhaps overkill. We want to prevent common errors, and we're not that concerned with atomicity between multiple processes - users should not be writing to the same CAR file concurrently.

cc @masih @willscott

貢獻者指南