createStoreMutex is not mutually exclusive: tryLock's callback means "retry", not "granted"

Start with packages/plugin/src/util/mutex.js and compare its tryLock callback handling with the documented storage-engine behavior. Update the mutex tests and their fakes in test/mutex.test.js and test/runState.test.js, then run the full plugin suite and confirm queued callers do not overlap in the provided concurrency scenarios.

TL;DR

• createStoreMutex resolves lock() when tryLock's callback fires, as if the lock had been handed over. Neither storage engine hands locks over. unlock erases the lock and fires every queued callback, so every woken waiter, plus any caller that arrives in the gap, runs the critical section at once.
• Reproduced on real lmdb 3.5.6 and @harperfast/rocksdb-js 2.8.0 (observed): up to 7 concurrent holders with 8 worker threads. A claim simulation built on the plugin's own runClaimPass and renderLease.js handed out 22–71 of 800 due URLs more than once across 8 runs.
• Scope: every release since prerender-v0.2.0 (mutex.js is unchanged since that tag), on both engines. It guards claim, setPause, refreshQueueStatus (which also resets the claim floor and reconciles the lease gauge), resetClaimFloor, runState and visitFilter.
• Fix: retry tryLock from the callback and resolve only when it returns true. The test fakes model hand-off, which is why the suite passes today. A patch with corrected fakes is ready (below); the full plugin suite passes with it.

Mechanism

The mutex (mutex.js#L23-L31):

new Promise((resolve) => {
	if (store.tryLock(lockKey, resolve)) resolve();
});

Its header (L10) says unlock "grants it to the next queued waiter". Both engines do something else (verified in their source):

The rocksdb-js README documents the callback as onUnlocked, a signal "to retry acquiring the lock".

The failure sequence:

1. A holds the lock. B and C call lock(); tryLock returns false and queues their callbacks.
2. A calls unlock(). The lock is erased and both callbacks fire, so B and C both resolve and run.
3. D calls lock() while they run. tryLock finds the key free and succeeds, so D runs too.
4. B finishes and calls unlock(), which releases the lock D is holding (unlock is keyed by key, not by owner).

A single woken waiter never overlaps the caller that just released. Overlap needs two or more waiters, or an arrival while the woken waiter is running.

Why the tests pass: the fakes in test/mutex.test.js and test/runState.test.js hand the lock to the next waiter on unlock (shift the queue, call that one callback), which neither engine does.

Reproduction

Engine semantics, observed identically on both engines:

const fired = [];
store.tryLock('k', () => fired.push('A')); // true (callback never fires)
store.tryLock('k', () => fired.push('B')); // false, queued
store.tryLock('k', () => fired.push('C')); // false, queued
store.unlock('k');
// 50 ms later: fired = ['B', 'C'], store.hasLock('k') === false

Mutex harness. This runs the plugin's createStoreMutex (byte-identical copy) against a real store shared by worker_threads. Each caller runs 150 iterations. Every scenario runs in three critical-section modes: a 1 ms timer await, one setImmediate turn, and a synchronous spin. The table shows the timer mode, which is closest to claim because claim awaits inside the lock. Environment: Node 24.15, macOS arm64.

• Ranges span both engines.
• "Unlocks by a caller that never acquired the lock" includes both releasing a key another caller held and releasing an already-free key.
• The fixed mutex held one holder, lost no updates and had no such unlocks in all 15 scenario × mode configurations on each engine.
• On the current mutex, the only runs without overlap were ones where the critical section did little or no awaiting.

Claim level. Worker threads run the plugin's runClaimPass (diff-checked verbatim against this commit), renderLease.js and hash.js, with the lease table in the real store's shared buffer, under the mutex. Two parts are modeled, not the plugin's:

• The index read is an in-memory sorted corpus that yields to the event loop every 50 rows.
• The wrapper is a harness loop, not RenderQueue.claim or claimSchedules.

8 threads × 10 claims over 800 due rows:

A duplicate grant happens when a concurrent pass's grant() finds the other pass's slot for the same key, treats it as a renewal and returns true. In most runs, total grants minus 800 exceeds the duplicate-URL count, so some URLs went out three or more times.

Impact per critical section

When it triggers (hypothesis)

• Steady state: rare. Overlap needs two or more queued waiters, or a waiter plus a new arrival while the woken one runs, all within a short claim pass. Pass durations were not measured.

• Bursts:

  • An MQTT empty → queued wake sends the whole fleet at one node at nearly the same moment.
  • Fleet restarts.
  • Harper restarts, where the ready set is empty and claims fall back to the longer index scan.

  Expect a few duplicate grants per burst. That is likely hard to tell apart from the duplicates a rolling restart already produces.

• Nothing in production has been traced to this yet. Grant races can hit the "publish CAS lost a race" branch added in 07d0e12. It surfaces as [prerender] claim could not record a lease for a due row (RenderQueue.js#L1414-L1422).

  • In the claim simulations, 0–6 passes per run were refused a lease with the current mutex, and 0 with the fix.
  • The same line also fires for a full probe window, so a hit in the logs is suggestive, not proof.

Proposed fix

const lock = () =>
	new Promise((resolve, reject) => {
		const attempt = () => {
			// The callback runs from a native threadsafe function, so a throw here
			// (store closed during shutdown) would otherwise be an uncaught exception.
			try {
				if (store.tryLock(lockKey, attempt)) resolve();
			} catch (error) {
				reject(error);
			}
		};
		attempt();
	});

• Not FIFO, and every release wakes all waiters. That cost 1.0–11.3 tryLock calls per acquisition across the harness configurations, and no starvation was observed. Fairness was not measured.
• Native withLock rejected: only rocksdb-js has it (lmdb 3.5.6 does not), and it shares the per-key queue with tryLock, so the two can't be mixed on one key.
• Tests:
  • Replace the hand-off fakes in test/mutex.test.js and test/runState.test.js with wake-all fakes. Otherwise runState's "two simultaneous starts" test times out with the fix; confirmed at an 8 s timeout.
  • mutex.test.js + runState.test.js with the new fakes: 5 of 24 fail against the current mutex, 24 of 24 pass with the fix.
  • Full plugin suite: 1412/1412 passing with the fix.
• A patch covering those three files applies cleanly to this commit, and prettier and eslint pass on it. I can open it as a PR.

Open items

• Untested:
  • The ready-set claim path.
  • Gauge reconcile racing claims.
  • Any run on Linux or inside a live Harper process; the harness drives the stores directly.
• Hypothesis, not tested, separate from this fix: the claimRun and setPause writes may join the request's ambient transaction and still be uncommitted when the lock is released. If so, no lock can serialize them.
• Related: rocksdb-js#848, a queued tryLock callback from a terminated worker aborting the process on unlock under Node 22; now closed. The retry loop re-queues callbacks more often, so confirm the pinned rocksdb-js includes that fix.
• No production log search has been done yet.