kvserver/intentresolver: persist partial progress when MVCC GC resolves a transaction record's lock spans

Open
#176,186 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
go

Research direction

Read pkg/kv/kvserver/intentresolver/intent_resolver.go around CleanupTxnIntentsOnGCAsync and resolveIntents, then trace the GC caller in pkg/kv/kvserver/gc/gc.go and the record update in pkg/kv/kvserver/batcheval/cmd_end_transaction.go. Determine how to persist each chunk only after its resolution responses succeed while avoiding lost updates from concurrent resolvers. Done means successful chunks are removed from the finalized record, remaining spans persist after failures, and MVCCStats stay correct.

Written by the indexing model from the issue text.

Description

A-kv-transactions C-enhancement O-agent O-support T-kv

Is your feature request related to a problem? Please describe.

When MVCC GC finds a finalized transaction record with a non-empty LockSpans, it hands the whole record to CleanupTxnIntentsOnGCAsync (gc.go:1246, intent_resolver.go:745). The record is deleted only after every lock span has resolved. If any batch fails, resolveIntents returns on the first error (intent_resolver.go:1175), the record is left untouched, and the next GC pass starts over from the first span. Nothing about the attempt survives.

#166156 bounded the concurrency of this work (chunks of 1000 per caller, a node-wide budget of 1000 in-flight intents), which stops the goroutine storm from #166037. It did not change the unit of success. For records with tens of thousands of lock spans every pass pays the full cost again, and with a 1 minute per-batch send timeout one slow batch wastes the pass. The queue admits at most 1000 such records per pass, and the large-abort-span requeue waits out kv.gc.txn_cleanup_threshold between passes (mvcc_gc_queue.go:521), so a range with 20k such records needs more than 21 consecutive perfect passes.

We have seen this three times in production: hundreds of records with 60k to 80k spans each, and most recently about 20k records averaging roughly 13k spans. In every case the records were removed by hand.

Describe the solution you'd like

After each chunk of a record's lock spans resolves successfully, rewrite the record with those spans removed, and delete it when none remain. A later pass then resumes from the unresolved remainder instead of from zero.

There is precedent for LockSpans on a finalized record meaning "the spans still to resolve": EndTxn resolves the locks local to its range synchronously and writes the record with only the external spans, or deletes it outright when none remain (cmd_end_transaction.go:851 through L864).

Safety questions to settle before implementing

  • GC only resolves records that are already finalized, or that it has first pushed to ABORTED because they are expired. Transaction recovery acts on STAGING records and reads InFlightWrites, not LockSpans (cmd_recover_txn.go:203). Confirm nothing else depends on a finalized record's LockSpans being complete.
  • A span may be dropped only once its resolution has been applied, not merely accepted by the batcher, so the rewrite must follow the responses for that chunk.
  • Concurrent resolvers (a reader that pushed the same transaction, or EndTxn cleanup still in flight) may be resolving the same spans. Resolving an already-resolved intent is a no-op, so this is a liveness question rather than a correctness one, but the record rewrite must not resurrect spans another writer removed. A conditional write on the record, or a dedicated command evaluated on the record's range that subtracts spans, would avoid lost updates.
  • The rewrite is a Raft write per chunk on the range holding the record. Chunk size should be large enough that this is negligible against the resolution work it protects.
  • MVCCStats for the record must be adjusted on each rewrite, as EndTxn already does.

Describe alternatives you've considered

  • Coalesce adjacent point spans into ResolveIntentRange requests before sending. Records produced by epoch retries of range-scanning statements have highly contiguous spans, so this cuts request counts by orders of magnitude, but it does not change restart semantics and is complementary.
  • Delete the record after a bounded number of failed passes and rely on lazy resolution when a reader next encounters an orphaned intent. Simpler, but it trades a bounded background cost for unbounded foreground latency on whoever touches those keys.
  • Record a resume index rather than trimming spans. Equivalent in effect, but it makes the record's meaning depend on a stable ordering of LockSpans, which trimming does not.

Additional context

This would also make a backport more attractive than #166156 alone for clusters that cannot upgrade, since it turns a cleanup that must succeed in one pass into one that converges across passes.

Jira issue: CRDB-70561

Dominant language
Go
Stars
32.6k
Forks
4.2k
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cockroachdb/cockroach

All issues in cockroachdb/cockroach

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.