sourceApply coordinated-retry loop has no backoff — hot spin re-encodes every write per round
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 76/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- node.js, typescript
- Domain
- backend, databases, performance
Research direction
Start in resources/DatabaseTransaction.ts at the coordinated-retry branch around lines 658-675, then compare it with the ERR_BUSY backoff and inspect the retry work around lines 496-500. Done means coordinated retries use the same bounded delay while preserving never-drop behavior; also review the sourceApply exemption around line 994.
Written by the indexing model from the issue text.
Description
Summary
Source-applied (replication-apply) transactions deliberately never drop on conflict — correct, there is no resume path for a dropped write. But the coordinated-retry branch recurses into commit() with no delay() at all, unlike the ERR_BUSY branch which backs off up to MAX_RETRY_DELAY_MS (1s). See v5.2.1 resources/DatabaseTransaction.ts#L658-L675 — identical on main.
Once retries > 0, every write in the transaction is fully re-saved per round (#L496-L500): entry reload + decode, full merge/index-diff/audit-construction/encode, fresh options object and promise chain per recursion. The long-transaction monitor explicitly exempts sourceApply (#L994), so nothing bounds or surfaces the spin.
Impact
A sustained local writer on a hot replicated key can pin a worker thread in a zero-backoff allocation loop indefinitely (source-vs-source contention serializes via apply-loop backpressure; local-vs-source does not). While spun, the apply loop is wedged: all peer legs for that database pause holding their in-flight WS buffers (replication_maxPayload, default 100MB each), and the pause-stall watchdog eventually reconnects and redelivers the same conflict. Contributed to heap pressure in an internal 16-node cluster incident.
Suggested fix
Add the same bounded backoff the ERR_BUSY branch already has to the coordinated-retry branch — one delay(Math.min(this.retries * this.retries, MAX_RETRY_DELAY_MS)) before the recursion. Never-drop semantics are preserved; only the spin rate changes.
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 196
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/harper
-
Derived index logs [error] "backend has no durable cursor undefined" at startup on a brand-new index Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
HarperFast/harper#2696 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HarperFast/harper#2650 ·
-
Runtime guard on HierarchicalNavigableSmallWorld.search() options argument (follow-up to #2165) Open
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
HarperFast/harper#2611 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
HarperFast/harper#2547 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HarperFast/harper#2503 ·
All issues in HarperFast/harper
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/skills#96 ·
-
[Block] Latest Posts [Type] Bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sugarlabs/musicblocks#8847 ·