Do not re-send non-idempotent writes after ambiguous failures in the b2 package

Start with b2/backend.go's withBackoff and the write methods listed in the issue, then trace retry classification in base/base.go. Read b2/compose.go and the existing Compose test in b2/compose_extended_wire_test.go for finish cleanup behavior. Run the recording-server cases described in the issue; done means ambiguous writes are not resent, errors carry the known identifiers, and Writer.Close and Compose do not cancel after an ambiguous finish.

Context

This is a follow-up to Backblaze/blazer#167 (merged, closes #165). Since #167, base.Action returns Retry for 408 on non-upload calls and for 502 and 504 on all calls, in addition to 429, 500 and 503. Every control-plane call in the b2 package goes through withBackoff, which re-sends the request whenever base.Action returns Retry, so non-idempotent writes are now also sent again after 408, 502 and 504, as they already were after 500 and after a transport error once the request was sent. A 502 or 504 does not say whether B2 applied the write. When it did, the second attempt either repeats the effect or turns a success into an error:

• Client.CreateKey, Bucket.CreateKey: a second key is created; the caller receives only the second key's secret, and the first key stays active until deleted manually.
• Client.NewBucket: duplicate_bucket_name although the bucket was created.
• Writer and Bucket.Compose large-file start: an extra unfinished large file.
• Writer.Close (finish): an error although the object exists; with WithCancelOnError, a cancel call on a finished file.
• Bucket.Compose (finish): an error although the object exists, followed by a deferred cancel of the file it may already have finished.
• Bucket.Copy (documented at b2/copy.go:109) and Object.Hide: an extra version or hide marker.
• Object.Delete, Object.Cancel: an error although the operation succeeded.
• Bucket.Update: a 409 because the revision already moved.

None of these errors says the outcome is unknown, and none carries the identifiers needed to check. This issue asks that the retries be limited to calls that are safe to repeat.

Current behavior

At a5e0d13:

• b2/backend.go:1056-1082: withBackoff applies one policy to every method; it retries when base.Action(err) == base.Retry (b2/baseline.go:240-242).
• Writes wrapped in it: createBucket (b2/backend.go:329-349), createKey (:375-395), createKeyMultiBucket (:397-417), updateBucket (:450-458), startLargeFile (:518-538), hideFile (:688-708), copyFile (:710-727), deleteFileVersion (:786-794), finishLargeFile (:954-974), cancel (:976-984), Key.del (:1043-1048).
• b2/compose.go:237-274: Compose starts and finishes through the same policy; after a failed finish, the deferred cleanup (:241-248) cancels the large file. b2/compose_extended_wire_test.go:902 expects that cancel.
• Classification: base/base.go:94-100 (408 on non-upload calls, and 429, 500, 502, 503 and 504 on all calls, return Retry), :552-556 (post-send transport errors marked retryable), :213-224 (MaxRetries, 5 for these calls).

Reproduced at a5e0d13 with a recording httptest server that fails the first call once and then succeeds:

status   b2_create_key  b2_copy_file  b2_hide_file  b2_update_bucket  b2_delete_file_version
408      2, err=nil     2, err=nil    2, err=nil    2, err=nil        2, err=nil
500      2, err=nil     2, err=nil    2, err=nil    2, err=nil        2, err=nil
502      2, err=nil     2, err=nil    2, err=nil    2, err=nil        2, err=nil
504      2, err=nil     2, err=nil    2, err=nil    2, err=nil        2, err=nil

Before #167 (7888c5b), 408, 502 and 504 sent each call once and returned the error. With Compose, a 408, 500, 502 or 504 on the first b2_finish_large_file sends it again; if the second finish fails because the first one was applied (a 400 in the test), Compose returns that error and calls b2_cancel_large_file.

Expected behavior

A write that may have been applied is not repeated automatically, and the caller gets an error that says so, with enough context to check the result. Reads keep the retries from #167.

Acceptance criteria

• R1. A 500, 502 or 504 after an applied createKey, finishLargeFile, copyFile, hideFile, deleteFileVersion or createBucket is not re-sent; the caller gets the unknown-outcome error with the identifiers above.
• R2. A dial error before send is still retried, and reads still retry on 408, 500, 502, 503 and 504.
• R3. Writer.Close and Compose after an ambiguous finish return the unknown-outcome error and do not call b2_cancel_large_file.

In scope

Proposed change

1. Keep withBackoff for reads and idempotent updates; add a write policy that retries only when the request provably did not reach B2 (pre-send dial/TLS errors, 429/503), never after a post-send transport error or a bare 500, 502 or 504. If repeating b2_copy_part with the same part number is safe (not confirmed here), it keeps the retries from #167.
2. Return a typed error, for example *b2.UnknownOutcomeError, with Method and any known FileID, KeyName, BucketName or SourceFileID, usable with errors.As.
3. Add reconcile helpers: look up a key by name after an ambiguous CreateKey; call b2_get_file_info on the large file ID after an ambiguous finish; treat file_not_present on a repeated delete as success only when the first outcome was unknown.
4. In Compose, do not cancel after an unknown-outcome finish.
5. Keep current retries for b2_upload_file, b2_upload_part and reads.

This works best with a method-aware base.Action, so base can tell callers which failures are safe to retry for each method.

Out of scope

• Unrelated work outside the components and behavior described in In scope.

Verification

Run the repository regression checks with live credentials unset:

unset B2_APPLICATION_KEY_ID B2_APPLICATION_KEY B2_ACCOUNT_ID B2_ACCOUNT_KEY
go build ./...
go vet ./...
go test -race -count=1 ./b2/... ./base

• Script an applied non-idempotent write followed by 500/502/504 or an ambiguous transport failure. Assert the literal request count, unknown-outcome error/identifiers, and absence of cancel after ambiguous finish.
• Prove pre-send retry and read-retry controls still behave as specified; restore each independent regression after its red proof.
• Expected result: the checks above succeed and the observable outcomes in Acceptance criteria are demonstrated. If a check cannot run, report Could not run because: ...; if it starts but fails, report Could not complete because: .... A skipped check is not a pass.

Links

Backblaze/blazer#167 (merged, closes #165) added the 408, 502 and 504 retries described above. Backblaze/blazer#101 (open) covers re-authentication on 401 unauthorized, not 5xx or transport retries. Backblaze/blazer#145 (merged, closes #144) synchronized authorization state; a request is still sent again once after a 401. Backblaze/blazer#129 added Compose, and Backblaze/blazer#171 (merged, closes #164) added Compose options and kept its cancel after a failed finish.