Delivery I/O runs inside the claim transaction: pool starvation, idle-in-transaction kills cause infinite redelivery, MVCC bloat

未关闭
  • 哪里不对？
#133 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
22/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
kafka, kotlin, mysql, postgresql
领域
backend, databases, distributed-systems

调研方向

Start with the KDoc on CompositeMessageDeliverer, which acknowledges the open transaction, and trace where the claim, delivery and status update share one transaction. Check the hard-coded CONNECT_TIMEOUT and REQUEST_TIMEOUT and the concurrency setting against the pool size. Done means the maintainers have agreed on the lease-based claim design or the startup guard, with tests covering a delivery that outlives the server's idle timeout.

由索引模型根据 Issue 内容生成。

描述

enhancement

Severity: Medium-High (pool starvation, duplicate-delivery loop, DB bloat) · Area: processor transaction design · Found in: code review of main @ e5630a1

Problem

The whole cycle (claim with FOR UPDATE SKIP LOCKED → network delivery → status update) runs in one DB transaction, so a pooled connection sits idle in transaction for as long as the transport I/O takes:

  • HTTP: up to 10 s connect + 30 s request timeout (hard-coded CONNECT_TIMEOUT/REQUEST_TIMEOUT, not configurable)
  • Kafka: producer.flush() blocks up to delivery.timeout.ms (120 s default)

Concrete failure modes:

  1. Connection-pool starvation. concurrency accepts values up to 256, and each worker holds one connection for the full delivery. Nothing checks this against the pool size: Hikari's default is 10. With concurrency >= pool size and a slow downstream, okapi takes every connection and the application's own requests time out on getConnection().
  2. Infinite duplicate loop under server-side timeouts. With Postgres idle_in_transaction_session_timeout, or a MySQL wait_timeout/proxy idle timeout shorter than the delivery time, the server kills the session mid-delivery. The final update/commit fails, and every entry that was actually delivered reverts to PENDING with retries unchanged. It is delivered again on the next tick, killed again, and so on for as long as the downstream stays slow. retries never advances, so nothing ever reaches FAILED.
  3. MVCC bloat (Postgres). Long transactions hold back the xmin horizon on what is by design a very high-churn table (every row is inserted, then updated, then deleted), which degrades vacuum for the entire database.
  4. Combined with the MySQL gap-lock issue, publishers are blocked for the same duration.

The KDoc on CompositeMessageDeliverer acknowledges the open transaction, but the consequences above aren't documented or guarded against.

Suggested fix

  • Preferred: a lease-based claim. In a short transaction, mark rows PROCESSING with locked_until = now + leaseTimeout (or bump a claim_token), commit, deliver outside any transaction, then write results in a second short transaction guarded by the claim token. A reaper (or the claim query itself) reclaims rows whose lease has expired.
  • At minimum: fail fast or warn at startup when concurrency ≥ the pool's maximum size; make the HTTP timeouts configurable; document the required relationship between delivery timeouts and idle_in_transaction_session_timeout/lock-wait timeouts.
主要语言
Kotlin
星标
14
派生
0
平均合并
9 小时 54 分钟
30 天内合并 PR
11

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

softwaremill/okapi 的其他 Issue

查看 softwaremill/okapi 的全部 Issue

相似的 Issue

更多 Kotlin Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。