kvserver/intentresolver: persist partial progress when MVCC GC resolves a transaction record's lock spans
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- go
- Domain
- databases, distributed-systems
Research direction
Read pkg/kv/kvserver/intentresolver/intent_resolver.go around CleanupTxnIntentsOnGCAsync and resolveIntents, then trace the GC caller in pkg/kv/kvserver/gc/gc.go and the record update in pkg/kv/kvserver/batcheval/cmd_end_transaction.go. Determine how to persist each chunk only after its resolution responses succeed while avoiding lost updates from concurrent resolvers. Done means successful chunks are removed from the finalized record, remaining spans persist after failures, and MVCCStats stay correct.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
When MVCC GC finds a finalized transaction record with a non-empty LockSpans, it hands the whole record to CleanupTxnIntentsOnGCAsync (gc.go:1246, intent_resolver.go:745). The record is deleted only after every lock span has resolved. If any batch fails, resolveIntents returns on the first error (intent_resolver.go:1175), the record is left untouched, and the next GC pass starts over from the first span. Nothing about the attempt survives.
#166156 bounded the concurrency of this work (chunks of 1000 per caller, a node-wide budget of 1000 in-flight intents), which stops the goroutine storm from #166037. It did not change the unit of success. For records with tens of thousands of lock spans every pass pays the full cost again, and with a 1 minute per-batch send timeout one slow batch wastes the pass. The queue admits at most 1000 such records per pass, and the large-abort-span requeue waits out kv.gc.txn_cleanup_threshold between passes (mvcc_gc_queue.go:521), so a range with 20k such records needs more than 21 consecutive perfect passes.
We have seen this three times in production: hundreds of records with 60k to 80k spans each, and most recently about 20k records averaging roughly 13k spans. In every case the records were removed by hand.
Describe the solution you'd like
After each chunk of a record's lock spans resolves successfully, rewrite the record with those spans removed, and delete it when none remain. A later pass then resumes from the unresolved remainder instead of from zero.
There is precedent for LockSpans on a finalized record meaning "the spans still to resolve": EndTxn resolves the locks local to its range synchronously and writes the record with only the external spans, or deletes it outright when none remain (cmd_end_transaction.go:851 through L864).
Safety questions to settle before implementing
- GC only resolves records that are already finalized, or that it has first pushed to ABORTED because they are expired. Transaction recovery acts on STAGING records and reads
InFlightWrites, notLockSpans(cmd_recover_txn.go:203). Confirm nothing else depends on a finalized record'sLockSpansbeing complete. - A span may be dropped only once its resolution has been applied, not merely accepted by the batcher, so the rewrite must follow the responses for that chunk.
- Concurrent resolvers (a reader that pushed the same transaction, or
EndTxncleanup still in flight) may be resolving the same spans. Resolving an already-resolved intent is a no-op, so this is a liveness question rather than a correctness one, but the record rewrite must not resurrect spans another writer removed. A conditional write on the record, or a dedicated command evaluated on the record's range that subtracts spans, would avoid lost updates. - The rewrite is a Raft write per chunk on the range holding the record. Chunk size should be large enough that this is negligible against the resolution work it protects.
- MVCCStats for the record must be adjusted on each rewrite, as
EndTxnalready does.
Describe alternatives you've considered
- Coalesce adjacent point spans into
ResolveIntentRangerequests before sending. Records produced by epoch retries of range-scanning statements have highly contiguous spans, so this cuts request counts by orders of magnitude, but it does not change restart semantics and is complementary. - Delete the record after a bounded number of failed passes and rely on lazy resolution when a reader next encounters an orphaned intent. Simpler, but it trades a bounded background cost for unbounded foreground latency on whoever touches those keys.
- Record a resume index rather than trimming spans. Equivalent in effect, but it makes the record's meaning depend on a stable ordering of
LockSpans, which trimming does not.
Additional context
This would also make a backport more attractive than #166156 alone for clusters that cannot upgrade, since it turns a cleanup that must succeed in one pass into one that converges across passes.
Jira issue: CRDB-70561
- Dominant language
- Go
- Stars
- 32.6k
- Forks
- 4.2k
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from cockroachdb/cockroach
-
C-bug O-community X-blathers-untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
cockroachdb/cockroach#176245 · 1 comment ·
-
sql: pg_database.dattablespace is 0 but pg_tablespace.oid is 1663, so they no longer joinPossibly taken @1AbdullahButt claimed this 14 days ago. OpenC-bug O-community X-blathers-triaged X-move-internal
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
cockroachdb/cockroach#176228 · 1 comment ·
-
PG compat: overflow in integer power calculation triggers incompatible SQLSTATEMay be free again @darshpatel1052 claimed this 18 days ago, and no pull request is open. OpenC-bug O-community X-blathers-untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
cockroachdb/cockroach#176223 · 2 comments ·
-
util/log: bufferedSinkCloser.Close leaks a goroutine after timeoutPossibly taken @dajiaohuang claimed this 14 days ago. OpenC-bug O-community X-blathers-untriaged
Difficulty 1/5 Under an hour Newbie friendliness 84/100
cockroachdb/cockroach#176210 · 1 comment ·
-
C-bug O-community T-sql-queries X-blathers-triaged X-move-internal
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
cockroachdb/cockroach#176150 · 1 comment ·
All issues in cockroachdb/cockroach
Similar issues
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
bug ready-for-triage ready-to-code
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
RedHatInsights/insights-ingress-go#946 · 4 comments ·
Maintainers usually reply within 1 day