ipld/go-car

blockstore: OpenReadWrite should use atomic writes

オープン

#249 opened on 2021/09/30

 (1 件のコメント) (0 件のリアクション) (0 人の担当者)Go (54 件のフォーク)github user discovery
P3help wanted

Repository metrics

Stars
 (176 個のスター)
PR merge metrics
 (平均マージ 23h 30m) (30d で 12 merged PRs)

説明

Right now, OpenReadWrite writes directly to the destination file. This has multiple problems:

  • If we encounter some error mid-finalize, we may leave a corrupted file
  • If we encounter a write/flush error while finalizing, we may leave a partial file
  • If we're resuming on an existing file and writing in-place, we may corrupt the user's data (e.g. https://github.com/ipld/go-car/issues/247)

It would be much saner overall to instead write to a temporary copy next to the destination file (such as foo.car.tmp for foo.car), and once Finalize has finished with no error, we do a rename that should basically never fail.

This also means that we can teach OpenReadWrite to always remove foo.car.tmp when it's discarded, to ensure we don't leave unfinished business behind.

The method described above should be doable with just a bit of os and io glue. We could use https://pkg.go.dev/github.com/google/renameio#TempFile, though I think pulling in a library for this use case is perhaps overkill. We want to prevent common errors, and we're not that concerned with atomicity between multiple processes - users should not be writing to the same CAR file concurrently.

cc @masih @willscott

コントリビューターガイド