TsavoriteLog fast-commit recovery adopts a stale suffix after recovering and reusing the tail

Start with the NUnit reproduction in libs/storage/Tsavorite/cs/test/test.hlog/LogFastCommitTests.cs, then read RestoreLatestAsync and ScanForwardForCommit in the referenced TsavoriteLog.cs and TsavoriteLogScanIterator.cs. Run the focused dotnet test command from the issue to reproduce the four wrong-tail failures. Done means recovery preserves the replacement tail and cookie across all four cases and scans only the acknowledged prefix plus the replacement entry; the issue describes this as requiring a durable recovery-history boundary.

Describe the bug

TsavoriteLog in fast-commit mode can recover past a newly acknowledged tail into records retained from an abandoned pre-recovery suffix. A raw upstream NUnit reproduction preserves every earlier acknowledged byte and changes exactly one aligned 512-byte sector of an unacknowledged payload to its OLD image. First recovery correctly selects the acknowledged prefix. After writing and committing a replacement entry, second recovery adopts a stale inline commit and extends TailAddress / CommittedUntilAddress into the abandoned history.

This occurs with equal, decreasing, and increasing stale commit numbers. A temporary strict-increase check fixes the first two but still accepts stale #5 after replacement #4. A stronger case overwrites the previous intermediate marker and zeroes the entire remaining tail-page padding; the surviving records and #5 marker form a valid-looking continuation of the replacement history.

Steps to reproduce the bug

Use upstream commit a1c98627220fbafde966191fe52b4fa5e3ee0eb0 (also fetched main during this investigation), with an NUnit test in libs/storage/Tsavorite/cs/test/test.hlog/LogFastCommitTests.cs and:

var device = new RandomAccessLocalStorageDevice(filename,
    disableFileBuffering: false, osReadBuffering: true);
var manager = new DeviceLogCommitCheckpointManager(
    new LocalStorageNamedDeviceFactoryCreator(),
    new DefaultCheckpointNamingScheme(directory),
    removeOutdated: false, fastCommitThrottleFreq: 256);
var log = new TsavoriteLog(new TsavoriteLogSettings
{
    LogDevice = device,
    LogCommitManager = manager,
    LogChecksum = LogChecksumType.PerEntry,
    FastCommitMode = true,
    TryRecoverLatest = false,
    PageSizeBits = 12,
    MemorySizeBits = 14,
    SegmentSizeBits = 14
});

1. Enqueue and strongly commit three 100-byte entries. Use deterministic random payload bytes (new Random(42).NextBytes(prefixEntry)) and no cookies. The acknowledged tail is 556, and the throttled metadata anchor is commit 1. Capture this OLD data image and the anchor metadata. The data file ends at 1024.
2. Enqueue a 2048-byte payload at 556 (new Random(123).NextBytes(payload)). For the increasing-number variants, issue strong commit 4 here without waiting; its inline marker starts at 2616, within page 0.
3. Enqueue two 3500-byte payloads (new Random(456).NextBytes(suffixEntry)). They land at 4096 and 8192. Issue the final strong commit: 4 without step 2's intermediate commit, or 5 with it. Its tail is 11756. Multiple outstanding commit requests are supported by the existing API.
4. Complete the NEW image solely for deterministic fault construction, then close the log/device. Construct a crash image by restoring exactly [1024,1536) to the OLD absent/zero sector. Every other byte comes from NEW. The modeled crash acknowledges only the prefix from step 1; completing the fixture's NEW image is not an application acknowledgment in that model.
5. Assert the tear is one sector, aligned, wholly inside the unacknowledged payload, and that every byte below 556 and the metadata anchor are unchanged. Reopen with fresh log/device/manager instances and call RecoverAsync(). Assert both tails are 556, and a scan returns exactly the three original entries.
6. Enqueue a replacement payload (new Random(789).NextBytes(replacementEntry)), strongly commit with cookie {2,4,6,8}, and close cleanly. Fresh manager state writes this replacement's metadata anchor. Use the variants below.
7. Reopen and assert the latest metadata anchor's UntilAddress is the replacement tail. Call RecoverAsync() again. The actual recovered tail is 11756 in every variant, instead of the replacement tail.

Raw NUnit case	Replacement	Stale marker	Expected tail	Actual tail
FastCommitRecoveryRetainsNewTailWithEqualStaleCommit	4-byte entry, commit #4	old #4	628	11756
FastCommitRecoveryRetainsNewTailWithDecreasingStaleCommit	4-byte entry, commit #6	old #4	628	11756
FastCommitRecoveryRetainsNewTailWithIncreasingStaleCommit	4-byte entry, commit #4	old #5	628	11756
FastCommitRecoveryRetainsNewTailAfterFullTailPageRewrite	3200-byte entry, commit #4	old #5	3824	11756

In the last case, the replacement's aligned write covers the remainder of page 0 and overwrites old commit 4. The test asserts that every byte [3824,4096) is zero. Each retained entry occupies 3512 bytes including its header and genuinely requires the next-page allocation. Therefore checking complete padding or requiring exactly the next commit number does not establish a recovery-history boundary.

Expected behavior

After recovery selects a prefix and a new commit extends it, subsequent recovery preserves that new history. For the cases above, both recovered tails equal the replacement's committed tail, the recovered cookie is {2,4,6,8}, and scanning yields exactly the three acknowledged prefix entries plus the replacement entry.

Screenshots

N/A; deterministic NUnit/file-image reproduction.

Release version

Source revision a1c98627220fbafde966191fe52b4fa5e3ee0eb0, .NET 10, Debug. Production source remained byte-for-byte unchanged after the investigation.

IDE

N/A; dotnet test CLI.

OS version

Windows. The tear model preserves whole 512-byte sectors as OLD or NEW. This is fault-model evidence rather than certification from a physical power-cut experiment.

Additional context

Relevant contracts and implementation:

• CommitInternal serializes marker creation, supports outstanding commits, and permits arbitrary increasing caller-supplied numbers.
• RestoreLatestAsync loads a metadata anchor, scans forward, and restores/reuses the selected tail and commit sequence.
• RecoveryReset clears the in-memory tail-page suffix. Later device pages remain available to a future recovery scan.
• ScanForwardForCommit accepts encountered valid commit metadata; its iterator skips zero length fields away from page start as padding.
• Version-1 commit metadata carries BeginAddress, UntilAddress, CommitNum and Cookie, with checksums. A complete correction needs a durable way to distinguish the current history from an abandoned suffix, such as crash-safe suffix invalidation or a persisted incarnation/predecessor relation. No new format or device API was implemented in this investigation.

Exact focused validation:

dotnet test libs\storage\Tsavorite\cs\test\test.hlog\Tsavorite.test.hlog.csproj -f net10.0 -c Debug --no-restore --filter "(FullyQualifiedName~FastCommitRecoveryRetainsNewTail|FullyQualifiedName~TsavoriteLogSimpleFastCommitTest|FullyQualifiedName~FastCommitRecoverToMissingCommitNumThrows)&Name!~EmulatedAzure" --logger "trx;LogFileName=retained-suffix-final-baseline.trx" --verbosity quiet

Pristine-source baseline with the four new cases: 4 passed, 4 failed, 0 skipped, exit 1. All four failures are the wrong recovered-tail assertions in the table.

Temporary strict-increase candidate (candidate.CommitNum <= info.CommitNum stops the scan before publishing candidate metadata): 6 passed, 2 failed, 0 skipped, exit 1. The two increasing-number cases still fail; equal/decreasing cases pass their tail, cookie and exact-data assertions. The candidate was removed as incomplete.

Existing controls pass in both runs: FastCommitRecoverToMissingCommitNumThrows, and TsavoriteLogSimpleFastCommitTest for LSD, MLSD and LocalMemory (including legitimate commits 1, 2 and 6). Restored pristine source was rebuilt and the four existing controls passed again, exit 0. Final TRX files were parsed and individual outcomes checked.

Searches found no matching issue or PR. #2065 addresses failed-recovery sentinel poisoning, #2076 addresses pre-listen recovery failure, and #2281 concerns device-level power-loss durability; this report isolates a raw log-history selection error under the specified sector-image model.