sqld: write lock can be handed to a connection that gave up, stalling all writes (plus two related lock-manager bugs)

Open
#2,286 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 11 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust, sqlite
Domain
backend, databases

Research direction

Start in libsql-server/src/connection/connection_manager.rs, tracing acquire(), close(), schedule_next(), and the lock-stealing path described in the report. Run codepilots/stress/stress.py against the supplied Docker image, then use the connection unit tests and concurrent stress harness to verify that writes do not stall, panic, or deadlock.

Written by the indexing model from the issue text.

Description

Summary

Under concurrent writes, sqld's write-lock scheduler (libsql-server/src/connection/connection_manager.rs) can hand the write lock to a connection that is no longer waiting for it. Nothing releases it afterwards: every writer then waits forever while busy-looping, all CPUs are pinned, and writes stop until the process is restarted (reads keep working). We hit this in production on ghcr.io/tursodatabase/libsql-server:latest (sqld 0.24.33, f8fb14f3); it took the site down for about an hour.

While fixing it we found two more bugs in the same file. The code is identical on current main (the file is unchanged since March 2025). Bug 1 reproduces reliably on the stock image; bug 2 we hit under stress testing; bug 3 we found by code review while chasing residual stalls under an extreme workload and have not caught in the act.

1. Lock handed to a connection that gave up

acquire() pushes the caller onto write_queue, then loops. Two early returns can happen after the push:

  • the slot is held by another connection's checkpoint → SQLITE_BUSY
  • the queue sync token changed → SQLITE_BUSY (517)

The queue entry stays behind. When schedule_next later pops it, the slot becomes Notified for a connection that is no longer in acquire() (frequently already closed). Nothing moves a Notified slot on; lock stealing only applies to Acquired. All other waiters then loop on park_deadline(slot.started_at + TXN_TIMEOUT), which is already in the past, so it returns immediately: a busy-spin per waiter.

The common trigger is a WAL auto-checkpoint after a commit: the checkpoint enqueues, finds another checkpoint holding the slot, returns SQLITE_BUSY, and the HTTP stream then closes. A typical precursor in the logs is begin_write_txn ... failed to rollback: cannot rollback - no transaction is active.

Lock-manager debug log from a reproduction (connection 1921):

checkpoint{id=1921}: enqueued                       # auto-checkpoint after its write
close{id=1921}: closed in 14µs                      # gave up with SQLITE_BUSY; stream closed
checkpoint{id=1923}:release:schedule_next: unparking id=1921
begin_write_txn{id=...}: current slot: Slot { id: 1921, ..., state: Notified }   # forever
2. Abort in close()

close() checks that it owns the slot, drops the guard, then calls release(). A slot in Failure state can be handed to the next waiter in between; release() then fails assert_eq!(slot.id, self.id) inside a SQLite callback and the process aborts (exit 133).

3. Deadlock when stealing the lock

The steal path drops the slot guard and then blocks on the owner's connection mutex to force a rollback. If the owner has meanwhile finished and is parked in acquire() again (e.g. its auto-checkpoint) while holding its own connection mutex, and the slot is handed to the stealer, each waits on the other forever.

Reproduction

With 40 clients upserting a single row, each request on its own /v2/pipeline stream that closes immediately, plus a couple of bulk writers that grow the WAL, libsql-server:latest stops accepting writes in every run, usually within 30–90 s. Script: codepilots/stress/stress.py (standard library only):

docker run -d -p 18080:8080 -e SQLD_NODE=primary ghcr.io/tursodatabase/libsql-server:latest
DURATION=180 python3 stress.py   # exits 2 when writes stall
Fixes

All three are fixed on the codepilots branch of our fork, one commit each, against f8fb14f3:

  • 8d47b6606 queue entries carry a cancelled flag; acquire() sets it (and passes on a slot already handed to it) when it gives up; schedule_next skips cancelled entries; waiters never busy-spin.
  • d1682fe72 close() checks and releases under one guard.
  • 77d038f28 the stealer only try_locks the owner and re-checks it still holds the same slot before rolling back; otherwise it backs off 10 ms.

With these, the reproduction and a harsher harness (150 writers, clients hanging up mid-transaction, transactions held past the 5 s steal timeout, at 1–4 CPUs) ran without a stall, panic or torn transaction across 25+ runs, plus a 60-minute soak, kill -9 and disk-full under load. The connection:: unit tests pass; test_many_concurrent fails intermittently at the same rate as on main. Details: CHANGELOG.md.

Happy to open a PR against main if that's useful.

Dominant language
C
Stars
17.3k
Forks
562
Avg merge
10d 15h
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tursodatabase/libsql

All issues in tursodatabase/libsql

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.