sqld: write lock can be handed to a connection that gave up, stalling all writes (plus two related lock-manager bugs)
I maintainer di solito rispondono entro 11 giorni
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 35/100
Direzione di ricerca
Start in libsql-server/src/connection/connection_manager.rs, tracing acquire(), close(), schedule_next(), and the lock-stealing path described in the report. Run codepilots/stress/stress.py against the supplied Docker image, then use the connection unit tests and concurrent stress harness to verify that writes do not stall, panic, or deadlock.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
Under concurrent writes, sqld's write-lock scheduler (libsql-server/src/connection/connection_manager.rs) can hand the write lock to a connection that is no longer waiting for it. Nothing releases it afterwards: every writer then waits forever while busy-looping, all CPUs are pinned, and writes stop until the process is restarted (reads keep working). We hit this in production on ghcr.io/tursodatabase/libsql-server:latest (sqld 0.24.33, f8fb14f3); it took the site down for about an hour.
While fixing it we found two more bugs in the same file. The code is identical on current main (the file is unchanged since March 2025). Bug 1 reproduces reliably on the stock image; bug 2 we hit under stress testing; bug 3 we found by code review while chasing residual stalls under an extreme workload and have not caught in the act.
1. Lock handed to a connection that gave up
acquire() pushes the caller onto write_queue, then loops. Two early returns can happen after the push:
- the slot is held by another connection's checkpoint →
SQLITE_BUSY - the queue sync token changed →
SQLITE_BUSY(517)
The queue entry stays behind. When schedule_next later pops it, the slot becomes Notified for a connection that is no longer in acquire() (frequently already closed). Nothing moves a Notified slot on; lock stealing only applies to Acquired. All other waiters then loop on park_deadline(slot.started_at + TXN_TIMEOUT), which is already in the past, so it returns immediately: a busy-spin per waiter.
The common trigger is a WAL auto-checkpoint after a commit: the checkpoint enqueues, finds another checkpoint holding the slot, returns SQLITE_BUSY, and the HTTP stream then closes. A typical precursor in the logs is begin_write_txn ... failed to rollback: cannot rollback - no transaction is active.
Lock-manager debug log from a reproduction (connection 1921):
checkpoint{id=1921}: enqueued # auto-checkpoint after its write
close{id=1921}: closed in 14µs # gave up with SQLITE_BUSY; stream closed
checkpoint{id=1923}:release:schedule_next: unparking id=1921
begin_write_txn{id=...}: current slot: Slot { id: 1921, ..., state: Notified } # forever
2. Abort in close()
close() checks that it owns the slot, drops the guard, then calls release(). A slot in Failure state can be handed to the next waiter in between; release() then fails assert_eq!(slot.id, self.id) inside a SQLite callback and the process aborts (exit 133).
3. Deadlock when stealing the lock
The steal path drops the slot guard and then blocks on the owner's connection mutex to force a rollback. If the owner has meanwhile finished and is parked in acquire() again (e.g. its auto-checkpoint) while holding its own connection mutex, and the slot is handed to the stealer, each waits on the other forever.
Reproduction
With 40 clients upserting a single row, each request on its own /v2/pipeline stream that closes immediately, plus a couple of bulk writers that grow the WAL, libsql-server:latest stops accepting writes in every run, usually within 30–90 s. Script: codepilots/stress/stress.py (standard library only):
docker run -d -p 18080:8080 -e SQLD_NODE=primary ghcr.io/tursodatabase/libsql-server:latest
DURATION=180 python3 stress.py # exits 2 when writes stall
Fixes
All three are fixed on the codepilots branch of our fork, one commit each, against f8fb14f3:
8d47b6606queue entries carry a cancelled flag;acquire()sets it (and passes on a slot already handed to it) when it gives up;schedule_nextskips cancelled entries; waiters never busy-spin.d1682fe72close()checks and releases under one guard.77d038f28the stealer onlytry_locks the owner and re-checks it still holds the same slot before rolling back; otherwise it backs off 10 ms.
With these, the reproduction and a harsher harness (150 writers, clients hanging up mid-transaction, transactions held past the 5 s steal timeout, at 1–4 CPUs) ran without a stall, panic or torn transaction across 25+ runs, plus a 60-minute soak, kill -9 and disk-full under load. The connection:: unit tests pass; test_many_concurrent fails intermittently at the same rate as on main. Details: CHANGELOG.md.
Happy to open a PR against main if that's useful.
- Lingua principale
- C
- Stelle
- 17.3k
- Fork
- 562
- Merge medio
- 10g 15h
- PR unite (30g)
- 1
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di tursodatabase/libsql
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
tursodatabase/libsql#2320 ·
I maintainer di solito rispondono entro 11 giorni
-
`cloudflare` feature: hrana streams are never closedForse già presa @pcbeingused333 l’ha presa 4 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
tursodatabase/libsql#2312 ·
I maintainer di solito rispondono entro 11 giorni
-
bottomless: finished S3 upload tasks are never reaped, so sqld memory grows by ~10 KB per WAL batchForse già presa @pavanmanishd l’ha presa 25 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
tursodatabase/libsql#2298 · 1 commento ·
I maintainer di solito rispondono entro 11 giorni
-
Rust local Row::get_value truncates embedded NUL TEXT and panics on malformed UTF-8 (0.9.30)Forse già presa @Sanjays2402 l’ha presa 35 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
tursodatabase/libsql#2285 ·
I maintainer di solito rispondono entro 11 giorni
-
libsql-ffi: broken builds under Nix on macOSForse già presa @frectonz l’ha presa 85 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
tursodatabase/libsql#2275 ·
I maintainer di solito rispondono entro 11 giorni
Tutte le issue di tursodatabase/libsql
Issue simili
-
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
JulianKemmerer/PipelineC#395 ·
-
upstream-blocked
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
libretro/parallel-n64#858 ·