sourceApply coordinated-retry loop has no backoff — hot spin re-encodes every write per round

Open Beginner friendly
#2,141 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
node.js, typescript

Research direction

Start in resources/DatabaseTransaction.ts at the coordinated-retry branch around lines 658-675, then compare it with the ERR_BUSY backoff and inspect the retry work around lines 496-500. Done means coordinated retries use the same bounded delay while preserving never-drop behavior; also review the sourceApply exemption around line 994.

Written by the indexing model from the issue text.

Description

Summary

Source-applied (replication-apply) transactions deliberately never drop on conflict — correct, there is no resume path for a dropped write. But the coordinated-retry branch recurses into commit() with no delay() at all, unlike the ERR_BUSY branch which backs off up to MAX_RETRY_DELAY_MS (1s). See v5.2.1 resources/DatabaseTransaction.ts#L658-L675 — identical on main.

Once retries > 0, every write in the transaction is fully re-saved per round (#L496-L500): entry reload + decode, full merge/index-diff/audit-construction/encode, fresh options object and promise chain per recursion. The long-transaction monitor explicitly exempts sourceApply (#L994), so nothing bounds or surfaces the spin.

Impact

A sustained local writer on a hot replicated key can pin a worker thread in a zero-backoff allocation loop indefinitely (source-vs-source contention serializes via apply-loop backpressure; local-vs-source does not). While spun, the apply loop is wedged: all peer legs for that database pause holding their in-flight WS buffers (replication_maxPayload, default 100MB each), and the pause-stall watchdog eventually reconnects and redelivers the same conflict. Contributed to heap pressure in an internal 16-node cluster incident.

Suggested fix

Add the same bounded backoff the ERR_BUSY branch already has to the coordinated-retry branch — one delay(Math.min(this.retries * this.retries, MAX_RETRY_DELAY_MS)) before the recursion. Never-drop semantics are preserved; only the spin rate changes.


🤖 Investigated and filed by Claude (Fable) on Nathan's behalf

Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
1d 15h
Merged PRs (30d)
196

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HarperFast/harper

All issues in HarperFast/harper

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.