sqld: write lock can be handed to a connection that gave up, stalling all writes (plus two related lock-manager bugs)

未关闭
#2,286 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 11 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
rust, sqlite
领域
backend, databases

调研方向

Start in libsql-server/src/connection/connection_manager.rs, tracing acquire(), close(), schedule_next(), and the lock-stealing path described in the report. Run codepilots/stress/stress.py against the supplied Docker image, then use the connection unit tests and concurrent stress harness to verify that writes do not stall, panic, or deadlock.

由索引模型根据 Issue 内容生成。

描述

Summary

Under concurrent writes, sqld's write-lock scheduler (libsql-server/src/connection/connection_manager.rs) can hand the write lock to a connection that is no longer waiting for it. Nothing releases it afterwards: every writer then waits forever while busy-looping, all CPUs are pinned, and writes stop until the process is restarted (reads keep working). We hit this in production on ghcr.io/tursodatabase/libsql-server:latest (sqld 0.24.33, f8fb14f3); it took the site down for about an hour.

While fixing it we found two more bugs in the same file. The code is identical on current main (the file is unchanged since March 2025). Bug 1 reproduces reliably on the stock image; bug 2 we hit under stress testing; bug 3 we found by code review while chasing residual stalls under an extreme workload and have not caught in the act.

1. Lock handed to a connection that gave up

acquire() pushes the caller onto write_queue, then loops. Two early returns can happen after the push:

  • the slot is held by another connection's checkpoint → SQLITE_BUSY
  • the queue sync token changed → SQLITE_BUSY (517)

The queue entry stays behind. When schedule_next later pops it, the slot becomes Notified for a connection that is no longer in acquire() (frequently already closed). Nothing moves a Notified slot on; lock stealing only applies to Acquired. All other waiters then loop on park_deadline(slot.started_at + TXN_TIMEOUT), which is already in the past, so it returns immediately: a busy-spin per waiter.

The common trigger is a WAL auto-checkpoint after a commit: the checkpoint enqueues, finds another checkpoint holding the slot, returns SQLITE_BUSY, and the HTTP stream then closes. A typical precursor in the logs is begin_write_txn ... failed to rollback: cannot rollback - no transaction is active.

Lock-manager debug log from a reproduction (connection 1921):

checkpoint{id=1921}: enqueued                       # auto-checkpoint after its write
close{id=1921}: closed in 14µs                      # gave up with SQLITE_BUSY; stream closed
checkpoint{id=1923}:release:schedule_next: unparking id=1921
begin_write_txn{id=...}: current slot: Slot { id: 1921, ..., state: Notified }   # forever
2. Abort in close()

close() checks that it owns the slot, drops the guard, then calls release(). A slot in Failure state can be handed to the next waiter in between; release() then fails assert_eq!(slot.id, self.id) inside a SQLite callback and the process aborts (exit 133).

3. Deadlock when stealing the lock

The steal path drops the slot guard and then blocks on the owner's connection mutex to force a rollback. If the owner has meanwhile finished and is parked in acquire() again (e.g. its auto-checkpoint) while holding its own connection mutex, and the slot is handed to the stealer, each waits on the other forever.

Reproduction

With 40 clients upserting a single row, each request on its own /v2/pipeline stream that closes immediately, plus a couple of bulk writers that grow the WAL, libsql-server:latest stops accepting writes in every run, usually within 30–90 s. Script: codepilots/stress/stress.py (standard library only):

docker run -d -p 18080:8080 -e SQLD_NODE=primary ghcr.io/tursodatabase/libsql-server:latest
DURATION=180 python3 stress.py   # exits 2 when writes stall
Fixes

All three are fixed on the codepilots branch of our fork, one commit each, against f8fb14f3:

  • 8d47b6606 queue entries carry a cancelled flag; acquire() sets it (and passes on a slot already handed to it) when it gives up; schedule_next skips cancelled entries; waiters never busy-spin.
  • d1682fe72 close() checks and releases under one guard.
  • 77d038f28 the stealer only try_locks the owner and re-checks it still holds the same slot before rolling back; otherwise it backs off 10 ms.

With these, the reproduction and a harsher harness (150 writers, clients hanging up mid-transaction, transactions held past the 5 s steal timeout, at 1–4 CPUs) ran without a stall, panic or torn transaction across 25+ runs, plus a 60-minute soak, kill -9 and disk-full under load. The connection:: unit tests pass; test_many_concurrent fails intermittently at the same rate as on main. Details: CHANGELOG.md.

Happy to open a PR against main if that's useful.

主要语言
C
星标
17.3k
派生
562
平均合并
10 天 15 小时
30 天内合并 PR
1

环境准备

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

tursodatabase/libsql 的其他 Issue

查看 tursodatabase/libsql 的全部 Issue

相似的 Issue

更多 C Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。