sqld: write lock can be handed to a connection that gave up, stalling all writes (plus two related lock-manager bugs)
维护者通常 11 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start in libsql-server/src/connection/connection_manager.rs, tracing acquire(), close(), schedule_next(), and the lock-stealing path described in the report. Run codepilots/stress/stress.py against the supplied Docker image, then use the connection unit tests and concurrent stress harness to verify that writes do not stall, panic, or deadlock.
由索引模型根据 Issue 内容生成。
描述
Summary
Under concurrent writes, sqld's write-lock scheduler (libsql-server/src/connection/connection_manager.rs) can hand the write lock to a connection that is no longer waiting for it. Nothing releases it afterwards: every writer then waits forever while busy-looping, all CPUs are pinned, and writes stop until the process is restarted (reads keep working). We hit this in production on ghcr.io/tursodatabase/libsql-server:latest (sqld 0.24.33, f8fb14f3); it took the site down for about an hour.
While fixing it we found two more bugs in the same file. The code is identical on current main (the file is unchanged since March 2025). Bug 1 reproduces reliably on the stock image; bug 2 we hit under stress testing; bug 3 we found by code review while chasing residual stalls under an extreme workload and have not caught in the act.
1. Lock handed to a connection that gave up
acquire() pushes the caller onto write_queue, then loops. Two early returns can happen after the push:
- the slot is held by another connection's checkpoint →
SQLITE_BUSY - the queue sync token changed →
SQLITE_BUSY(517)
The queue entry stays behind. When schedule_next later pops it, the slot becomes Notified for a connection that is no longer in acquire() (frequently already closed). Nothing moves a Notified slot on; lock stealing only applies to Acquired. All other waiters then loop on park_deadline(slot.started_at + TXN_TIMEOUT), which is already in the past, so it returns immediately: a busy-spin per waiter.
The common trigger is a WAL auto-checkpoint after a commit: the checkpoint enqueues, finds another checkpoint holding the slot, returns SQLITE_BUSY, and the HTTP stream then closes. A typical precursor in the logs is begin_write_txn ... failed to rollback: cannot rollback - no transaction is active.
Lock-manager debug log from a reproduction (connection 1921):
checkpoint{id=1921}: enqueued # auto-checkpoint after its write
close{id=1921}: closed in 14µs # gave up with SQLITE_BUSY; stream closed
checkpoint{id=1923}:release:schedule_next: unparking id=1921
begin_write_txn{id=...}: current slot: Slot { id: 1921, ..., state: Notified } # forever
2. Abort in close()
close() checks that it owns the slot, drops the guard, then calls release(). A slot in Failure state can be handed to the next waiter in between; release() then fails assert_eq!(slot.id, self.id) inside a SQLite callback and the process aborts (exit 133).
3. Deadlock when stealing the lock
The steal path drops the slot guard and then blocks on the owner's connection mutex to force a rollback. If the owner has meanwhile finished and is parked in acquire() again (e.g. its auto-checkpoint) while holding its own connection mutex, and the slot is handed to the stealer, each waits on the other forever.
Reproduction
With 40 clients upserting a single row, each request on its own /v2/pipeline stream that closes immediately, plus a couple of bulk writers that grow the WAL, libsql-server:latest stops accepting writes in every run, usually within 30–90 s. Script: codepilots/stress/stress.py (standard library only):
docker run -d -p 18080:8080 -e SQLD_NODE=primary ghcr.io/tursodatabase/libsql-server:latest
DURATION=180 python3 stress.py # exits 2 when writes stall
Fixes
All three are fixed on the codepilots branch of our fork, one commit each, against f8fb14f3:
8d47b6606queue entries carry a cancelled flag;acquire()sets it (and passes on a slot already handed to it) when it gives up;schedule_nextskips cancelled entries; waiters never busy-spin.d1682fe72close()checks and releases under one guard.77d038f28the stealer onlytry_locks the owner and re-checks it still holds the same slot before rolling back; otherwise it backs off 10 ms.
With these, the reproduction and a harsher harness (150 writers, clients hanging up mid-transaction, transactions held past the 5 s steal timeout, at 1–4 CPUs) ran without a stall, panic or torn transaction across 25+ runs, plus a 60-minute soak, kill -9 and disk-full under load. The connection:: unit tests pass; test_many_concurrent fails intermittently at the same rate as on main. Details: CHANGELOG.md.
Happy to open a PR against main if that's useful.
- 主要语言
- C
- 星标
- 17.3k
- 派生
- 562
- 平均合并
- 10 天 15 小时
- 30 天内合并 PR
- 1
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
tursodatabase/libsql 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 68/100
tursodatabase/libsql#2320 ·
维护者通常 11 天内回复
-
`cloudflare` feature: hrana streams are never closed可能已有人在做 @pcbeingused333 于 4 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
tursodatabase/libsql#2312 ·
维护者通常 11 天内回复
-
bottomless: finished S3 upload tasks are never reaped, so sqld memory grows by ~10 KB per WAL batch可能已有人在做 @pavanmanishd 于 25 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 83/100
tursodatabase/libsql#2298 · 1 条评论 ·
维护者通常 11 天内回复
-
Rust local Row::get_value truncates embedded NUL TEXT and panics on malformed UTF-8 (0.9.30)可能已有人在做 @Sanjays2402 于 35 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
tursodatabase/libsql#2285 ·
维护者通常 11 天内回复
-
libsql-ffi: broken builds under Nix on macOS可能已有人在做 @frectonz 于 85 天前认领。 未关闭
难度 2/5 1-3 小时 新手友好度 86/100
tursodatabase/libsql#2275 ·
维护者通常 11 天内回复
查看 tursodatabase/libsql 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 92/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
JulianKemmerer/PipelineC#395 ·
-
upstream-blocked
难度 2/5 1-3 小时 新手友好度 65/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
-
难度 2/5 1-3 小时 新手友好度 70/100
libretro/parallel-n64#858 ·