sqld: write lock can be handed to a connection that gave up, stalling all writes (plus two related lock-manager bugs)

Aperta
#2,286 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 11 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
rust, sqlite
Ambito
backend, databases

Direzione di ricerca

Start in libsql-server/src/connection/connection_manager.rs, tracing acquire(), close(), schedule_next(), and the lock-stealing path described in the report. Run codepilots/stress/stress.py against the supplied Docker image, then use the connection unit tests and concurrent stress harness to verify that writes do not stall, panic, or deadlock.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Under concurrent writes, sqld's write-lock scheduler (libsql-server/src/connection/connection_manager.rs) can hand the write lock to a connection that is no longer waiting for it. Nothing releases it afterwards: every writer then waits forever while busy-looping, all CPUs are pinned, and writes stop until the process is restarted (reads keep working). We hit this in production on ghcr.io/tursodatabase/libsql-server:latest (sqld 0.24.33, f8fb14f3); it took the site down for about an hour.

While fixing it we found two more bugs in the same file. The code is identical on current main (the file is unchanged since March 2025). Bug 1 reproduces reliably on the stock image; bug 2 we hit under stress testing; bug 3 we found by code review while chasing residual stalls under an extreme workload and have not caught in the act.

1. Lock handed to a connection that gave up

acquire() pushes the caller onto write_queue, then loops. Two early returns can happen after the push:

  • the slot is held by another connection's checkpoint → SQLITE_BUSY
  • the queue sync token changed → SQLITE_BUSY (517)

The queue entry stays behind. When schedule_next later pops it, the slot becomes Notified for a connection that is no longer in acquire() (frequently already closed). Nothing moves a Notified slot on; lock stealing only applies to Acquired. All other waiters then loop on park_deadline(slot.started_at + TXN_TIMEOUT), which is already in the past, so it returns immediately: a busy-spin per waiter.

The common trigger is a WAL auto-checkpoint after a commit: the checkpoint enqueues, finds another checkpoint holding the slot, returns SQLITE_BUSY, and the HTTP stream then closes. A typical precursor in the logs is begin_write_txn ... failed to rollback: cannot rollback - no transaction is active.

Lock-manager debug log from a reproduction (connection 1921):

checkpoint{id=1921}: enqueued                       # auto-checkpoint after its write
close{id=1921}: closed in 14µs                      # gave up with SQLITE_BUSY; stream closed
checkpoint{id=1923}:release:schedule_next: unparking id=1921
begin_write_txn{id=...}: current slot: Slot { id: 1921, ..., state: Notified }   # forever
2. Abort in close()

close() checks that it owns the slot, drops the guard, then calls release(). A slot in Failure state can be handed to the next waiter in between; release() then fails assert_eq!(slot.id, self.id) inside a SQLite callback and the process aborts (exit 133).

3. Deadlock when stealing the lock

The steal path drops the slot guard and then blocks on the owner's connection mutex to force a rollback. If the owner has meanwhile finished and is parked in acquire() again (e.g. its auto-checkpoint) while holding its own connection mutex, and the slot is handed to the stealer, each waits on the other forever.

Reproduction

With 40 clients upserting a single row, each request on its own /v2/pipeline stream that closes immediately, plus a couple of bulk writers that grow the WAL, libsql-server:latest stops accepting writes in every run, usually within 30–90 s. Script: codepilots/stress/stress.py (standard library only):

docker run -d -p 18080:8080 -e SQLD_NODE=primary ghcr.io/tursodatabase/libsql-server:latest
DURATION=180 python3 stress.py   # exits 2 when writes stall
Fixes

All three are fixed on the codepilots branch of our fork, one commit each, against f8fb14f3:

  • 8d47b6606 queue entries carry a cancelled flag; acquire() sets it (and passes on a slot already handed to it) when it gives up; schedule_next skips cancelled entries; waiters never busy-spin.
  • d1682fe72 close() checks and releases under one guard.
  • 77d038f28 the stealer only try_locks the owner and re-checks it still holds the same slot before rolling back; otherwise it backs off 10 ms.

With these, the reproduction and a harsher harness (150 writers, clients hanging up mid-transaction, transactions held past the 5 s steal timeout, at 1–4 CPUs) ran without a stall, panic or torn transaction across 25+ runs, plus a 60-minute soak, kill -9 and disk-full under load. The connection:: unit tests pass; test_many_concurrent fails intermittently at the same rate as on main. Details: CHANGELOG.md.

Happy to open a PR against main if that's useful.

Lingua principale
C
Stelle
17.3k
Fork
562
Merge medio
10g 15h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tursodatabase/libsql

Tutte le issue di tursodatabase/libsql

Issue simili

Altre issue su C

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.