Pin Reader chunks to one file version, read listed versions by ID, and make Verify work for large files

Aperta
  • Qualcosa non va?
#180 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
api, backend, testing

Direzione di ricerca

Start with b2/reader.go, especially chunk downloading and Verify, then trace NewRangeReader in b2/b2.go and listed versions in b2/iterator.go. Run or extend the x/b2sim coverage; the issue notes its by-ID endpoint currently returns 501. Done means reads return one version or an error, listed versions read by ID, and Verify detects differing chunk hashes.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug enhancement

Summary

Reader downloads every chunk with b2_download_file_by_name, so each chunk comes from whatever version is current at that moment:

  1. If the object is overwritten during a read longer than one ChunkSize (10 MB by default), the returned bytes mix two versions with no error.
  2. Object.NewReader on an Object from List(ctx, ListHidden()), which represents one specific version, ignores that version's file ID and reads the latest version by name (or fails when the latest is a hide marker). The only by-ID download is the single-request Bucket.DownloadFileByID; a full Reader by ID is requested in Backblaze/blazer#158.
  3. Verify cannot catch either case for large files: Writer records large_file_sha1 only when the caller supplies Attrs.SHA1 (and, per Backblaze/blazer#169, drops it when the file info already has 10 entries), so B2 reports none and Verify returns (nil, false). When a SHA-1 is present, r.sha1 takes the value from the last chunk instead of comparing chunks.

Current behavior

At a5e0d13:

  • b2/reader.go:167: every chunk uses downloadFileByName; the response file ID (X-Bz-File-Id) is not read.
  • b2/reader.go:183-187: the expected hash follows the latest chunk.
  • b2/b2.go:763-782: NewRangeReader keeps only o.name, even when o.f holds a listed version (b2/iterator.go:271-279).
  • Download by ID exists below the Reader: base/base.go:1790-1801, b2/backend.go:666, and b2/b2.go:798-808 (Bucket.DownloadFileByID, one request).
  • b2/writer.go:777 and :784-796 (fileInfoFromAttrs): large_file_sha1 only from caller input, and only below 10 file-info entries; getLargeFile (b2/writer.go:407-418) starts the large file without it.
  • b2/reader.go:357-371: Verify returns (nil, false) without a 40-digit hash.

Reproduced locally at a5e0d13 with x/b2sim: an 8-byte object AAAAAAAA read with ChunkSize = 4, overwritten with BBBBBBBB after the first chunk. The reader returned AAAABBBB with no error. For this simple file the simulator sends a real SHA-1, so Verify() reported a bad hash; for a large file with no whole-file SHA-1 it returns (nil, false) and the mix goes unnoticed. Calling NewReader on the older version from List(ctx, ListHidden()) returned BBBBBBBB.

Expected behavior

A Reader returns the bytes of one version or an error, and a listed version can be read by ID.

Proposed change

  1. Record the file ID from the first chunk and fetch later chunks by that ID with the existing downloadFileByID, or fail if a later by-name response reports a different X-Bz-File-Id.
  2. When the Object carries a file ID (listing, Copy, finished Writer), download by ID from the first chunk; document that Bucket.Object(name) reads the latest version. This overlaps Backblaze/blazer#158.
  3. In Verify, compare every chunk's SHA-1 header with the first.
  4. Optionally compute and send large_file_sha1 (opt-in, or in ReadFrom for seekable sources).

Acceptance criteria

  • Overwriting the object between chunks yields the original content or an error, never a mix.
  • NewReader on a listed non-latest version returns that version's bytes.
  • Verify reports a mismatch when chunk SHA-1 headers disagree.
  • x/b2sim covers a download-during-overwrite case; it currently answers b2_download_file_by_id with 501 (x/b2sim/sim.go:7-12), so that endpoint is needed to test the by-ID path.

Related

Backblaze/blazer#158 (open, full Reader for DownloadFileByID) overlaps point 2. Backblaze/blazer#127 (merged) added DownloadFileByID. Backblaze/blazer#169 (open) keeps a caller-supplied large_file_sha1 regardless of the file-info count, which helps point 3. Backblaze/blazer#147 (merged) added x/b2sim.

Lingua principale
Go
Stelle
66
Fork
18
Merge medio
1g 10h
PR unite (30g)
132

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Backblaze/blazer

Tutte le issue di Backblaze/blazer

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.