Omnichannel queue: stale inquiry locks cause repeated takeInquiry failures after lock expiration

Read packages/models/src/models/LivechatInquiry.ts and trace findNextAndLock, takeInquiry, and unlock; then follow checkQueue in apps/meteor/server/services/omnichannel/queue.ts and takeInquiry in RoutingManager.ts. Start with the model tests and queue.test.ts described in the issue, covering stale lock reacquisition and ownership. Done means a reacquired inquiry can be taken and an expired worker cannot take or release another worker’s lock. An open linked pull request already addresses this issue.

Description:

When an Omnichannel inquiry still carries a lock from an earlier queue cycle, every later cycle fails to take it with error-taking-inquiry-lockedAt-mismatch, and the failure leaves a fresh lock behind, so the next cycle fails the same way. The inquiry stays queued while agents are available, until the queue worker is stopped or someone takes it manually.

Affected code (develop 45d9273ed3):

• packages/models/src/models/LivechatInquiry.ts
  • findNextAndLock sets locked/lockedAt with findOneAndUpdate but passes no returnDocument, so the driver (mongodb 6.16.0) returns the document before the update.
  • takeInquiry(inquiryId, lockedAt?) filters on lockedAt when one is given.
  • unlock(inquiryId) clears locked/lockedAt unconditionally.
  • queueInquiry puts an inquiry back to queued without touching locked/lockedAt.
• apps/meteor/server/lib/omnichannel/RoutingManager.ts, takeInquiry: calls LivechatInquiry.takeInquiry(_id, inquiry.lockedAt) and throws error-taking-inquiry-lockedAt-mismatch when nothing is modified.
• apps/meteor/server/services/omnichannel/queue.ts, checkQueue: locks with findNextAndLock, unlocks after processing, and does not unlock when processing throws.

Root cause

The lockedAt that checkQueue hands to RoutingManager.takeInquiry is the value from before the lock was taken, not the lock that worker now holds.

• Fresh inquiry: the old document has no lockedAt, so takeInquiry gets undefined, skips the filter, and succeeds. The comparison added in #36749 never applies here.
• Inquiry that still has a lock (T1) left from an earlier cycle: once T1 is older than INQUIRY_LOCK_TIMEOUT, findNextAndLock writes T2 and returns T1. takeInquiry({ _id, lockedAt: T1 }) matches nothing and the mismatch error is thrown. checkQueue catches it without unlocking, so T2 stays. When T2 expires, the same thing happens with T2/T3, and so on.

A stale lock is left behind when processing throws between lock and unlock, when the process stops in the middle of a cycle (unlockAll runs only when the queue worker is stopped, not on startup), or when assignment fails after the inquiry was marked taken and the inquiry is later re-queued with queueInquiry.

A related ownership problem: because unlock is unconditional, a worker whose lock has expired can clear the lock of the worker that re-acquired the inquiry.

Steps to reproduce:

Using LivechatInquiryRaw against a real MongoDB:

1. Insert an inquiry with status: 'queued'.
2. Call findNextAndLock(sort, null) and do not unlock (simulates a cycle that threw or a process that stopped).
3. Wait until lockedAt is older than INQUIRY_LOCK_TIMEOUT.
4. Call findNextAndLock(sort, null) again; it returns the previous lockedAt while a newer one is stored.
5. Call takeInquiry(id, returned.lockedAt): modifiedCount is 0, which RoutingManager.takeInquiry turns into error-taking-inquiry-lockedAt-mismatch.
6. Repeat 3–5: it fails every time.

Expected behavior:

After a stale lock expires, the next queue cycle that re-acquires the inquiry can take it. Only the worker that currently holds the lock can take or release it.

Actual behavior:

Every cycle after the first stale lock fails with error-taking-inquiry-lockedAt-mismatch and leaves a new lock behind. A worker whose lock has expired can still take the inquiry, while the worker that re-acquired it is rejected, and its unlock clears the other worker's active lock.

What is not affected: a normal cycle on an unlocked inquiry, and the "no agents available" path, which unlocks the inquiry so that it is picked up normally later. Other inquiries in the same queue keep being routed. The stuck inquiry only takes up a queue cycle each time its lock expires.

Reproduction output

I ran this with jest against a real mongod 7.0.24 (mongodb-memory-server), driver 6.16.0, using LivechatInquiryRaw from packages/models at 45d9273ed3. The checkQueue / RoutingManager.takeInquiry branches were reproduced step by step around the real model calls. INQUIRY_LOCK_TIMEOUT was set to 1500 ms to keep the run short; the code reads it from the environment. Run at 2026-11-04 22:47 UTC; command: npx jest -c <scratch>/jest.config.js --coverage=false --runInBand (exit code 0).

Interrupted cycle, then four cycles with an agent available:

cycle1: prev lockedAt=undefined | persisted now=2026-11-04T22:47:18.499Z | returned=undefined | passed to takeInquiry=undefined | worker dies before unlock
cycle1b (before expiry): findNextAndLock -> null (persisted locked=true lockedAt=2026-11-04T22:47:18.499Z)
cycle2: prev lockedAt=2026-11-04T22:47:18.499Z | persisted now=2026-11-04T22:47:20.221Z | returned=2026-11-04T22:47:18.499Z | passed to takeInquiry=2026-11-04T22:47:18.499Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:20.221Z
cycle3: prev lockedAt=2026-11-04T22:47:20.221Z | persisted now=2026-11-04T22:47:22.104Z | returned=2026-11-04T22:47:20.221Z | passed to takeInquiry=2026-11-04T22:47:20.221Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:22.104Z
cycle4: prev lockedAt=2026-11-04T22:47:22.104Z | persisted now=2026-11-04T22:47:23.816Z | returned=2026-11-04T22:47:22.104Z | passed to takeInquiry=2026-11-04T22:47:22.104Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:23.816Z
cycle5: prev lockedAt=2026-11-04T22:47:23.816Z | persisted now=2026-11-04T22:47:25.522Z | returned=2026-11-04T22:47:23.816Z | passed to takeInquiry=2026-11-04T22:47:23.816Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:25.522Z

Lock released with unlockAll() (what the queue worker does when stopped), after which the inquiry is taken:

cycle1: prev lockedAt=undefined | persisted now=2026-11-04T22:47:29.507Z | returned=undefined | passed to takeInquiry=undefined | worker dies before unlock
cycle2: prev lockedAt=2026-11-04T22:47:29.507Z | persisted now=2026-11-04T22:47:31.224Z | returned=2026-11-04T22:47:29.507Z | passed to takeInquiry=2026-11-04T22:47:29.507Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:31.224Z
unlockAll() called (OmnichannelQueue._stop)
cycle3: prev lockedAt=undefined | persisted now=2026-11-04T22:47:31.241Z | returned=undefined | passed to takeInquiry=undefined | result=TAKEN | after: status=taken locked=undefined

No agents: unlocked each time, taken once an agent is available (works correctly):

cycle1: prev lockedAt=undefined | persisted now=2026-11-04T22:47:31.274Z | returned=undefined | passed to takeInquiry=undefined | result=not taken (no agent), unlocked | after: status=queued locked=undefined
cycle2: prev lockedAt=undefined | persisted now=2026-11-04T22:47:31.282Z | returned=undefined | passed to takeInquiry=undefined | result=not taken (no agent), unlocked | after: status=queued locked=undefined
cycle3: prev lockedAt=undefined | persisted now=2026-11-04T22:47:31.286Z | returned=undefined | passed to takeInquiry=undefined | result=TAKEN | after: status=taken locked=undefined

Taken, assignment fails, then re-queued with queueInquiry: the lock survives the re-queue and the inquiry gets stuck:

after take + assignAgent throw: {"_id":"inq-5","status":"taken","locked":true,"lockedAt":"2026-11-04T22:47:31.308Z"}
after queueInquiry: {"_id":"inq-5","status":"queued","locked":true,"lockedAt":"2026-11-04T22:47:31.308Z"}
cycle2: prev lockedAt=2026-11-04T22:47:31.308Z | persisted now=2026-11-04T22:47:33.021Z | returned=2026-11-04T22:47:31.308Z | passed to takeInquiry=2026-11-04T22:47:31.308Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:33.021Z
cycle3: prev lockedAt=2026-11-04T22:47:33.021Z | persisted now=2026-11-04T22:47:34.732Z | returned=2026-11-04T22:47:33.021Z | passed to takeInquiry=2026-11-04T22:47:33.021Z | ERROR error-taking-inquiry-lockedAt-mismatch | after: status=queued locked=true lockedAt=2026-11-04T22:47:34.732Z

Two workers: concurrent acquisition is exclusive, but once A's lock expires and B re-acquires, the guard keeps the wrong worker, and A's unlock clears B's lock:

workerA got=true workerB got=false
A: returned lockedAt=undefined (persisted 2026-11-04T22:47:34.756Z)
B: returned lockedAt=2026-11-04T22:47:34.756Z (persisted 2026-11-04T22:47:36.459Z)
B (current owner) takeInquiry modified=0; A (expired owner) takeInquiry modified=1
B lock before A.unlock: locked=true lockedAt=2026-11-04T22:47:38.180Z; after: locked=undefined lockedAt=undefined
third worker C can now acquire while B is still processing: true

Same scenarios with returnDocument: 'after' applied to the findOneAndUpdate call (model source unchanged, option injected in the test):

cycle1: prev lockedAt=undefined | persisted now=2026-11-04T22:47:38.305Z | returned=2026-11-04T22:47:38.305Z | passed to takeInquiry=2026-11-04T22:47:38.305Z | worker dies before unlock
cycle2: prev lockedAt=2026-11-04T22:47:38.305Z | persisted now=2026-11-04T22:47:40.018Z | returned=2026-11-04T22:47:40.018Z | passed to takeInquiry=2026-11-04T22:47:40.018Z | result=TAKEN | after: status=taken locked=undefined
A returned=2026-11-04T22:47:40.212Z take modified=0; B returned=2026-11-04T22:47:41.923Z take modified=1
after expired A.unlock: locked=undefined

Test Suites: 1 passed, 1 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        36.7 s
Ran all test suites.

The last line in the second block shows that returnDocument: 'after' alone does not stop an expired worker from clearing another worker's lock.

Production impact and limitations

• Applies when the queue worker runs (routing methods with auto-assignment) and inquiries are queued, e.g. with the waiting queue enabled or when no agent was available when the chat arrived.
• It needs one of the triggers above to happen first. A normal cycle or a cycle with no agents available doesn't trigger it.
• Once triggered, the inquiry does not recover on its own. It recovers when the queue worker is stopped (routing method or Livechat setting change, which runs unlockAll) or when an agent takes it manually (that path reads the current lockedAt from the database).
• I have not measured how often the triggers happen in production and found no existing report of this symptom.

Suggested fix

1. Pass returnDocument: 'after' in findNextAndLock, so the caller gets the lock it actually holds (queueInquiry in the same model already does this).
2. Make unlock conditional on that lockedAt, so a worker can only release its own lock. Do not add an unconditional unlock to the error path, since it would clear another worker's lock. With (1), a lock left by a failed cycle simply expires and the next cycle takes the inquiry.

Regression test plan

Model tests against a real MongoDB (mongodb-memory-server, as ee/packages/abac already does), expiring a lock by moving lockedAt back in time instead of sleeping:

• fresh inquiry: lock returns the stored lockedAt, take succeeds;
• stale lock: re-acquire returns the new lockedAt and take succeeds (fails on current code);
• repeated attempts after several stale cycles;
• two concurrent findNextAndLock calls: only one acquires;
• expired worker A vs. re-acquiring worker B: A's take and A's unlock are rejected, B's succeed (both fail on current code);
• no-agent cycle unlocks with its own token;
• taken → assignment failure → queueInquiry → next cycle takes it.

And in queue.test.ts, check that checkQueue unlocks with the acquired lockedAt.

Server Setup Information:

• Version of Rocket.Chat Server: develop (9.1.0-develop, 45d9273ed3)
• License Type: any (Auto Selection in CE is affected)
• Number of Running Instances: any. One instance is enough after a mid-cycle stop. With several instances, a lock can also expire while another instance is still processing.
• NodeJS Version: v26.0.0 (test run)
• MongoDB Version: 7.0.24 (test run)

Additional context

• #36749 added the lockedAt comparison in takeInquiry and the configurable INQUIRY_LOCK_TIMEOUT.
• #37776 added agent-level locking, which is unrelated to this inquiry lock.