Delivery I/O runs inside the claim transaction: pool starvation, idle-in-transaction kills cause infinite redelivery, MVCC bloat

未關閉
  • 哪裡不對？
#133 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
5/5
預估耗時
一週以上
新手友好度
22/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
kafka, kotlin, mysql, postgresql
領域
backend, databases, distributed-systems

研究方向

Start with the KDoc on CompositeMessageDeliverer, which acknowledges the open transaction, and trace where the claim, delivery and status update share one transaction. Check the hard-coded CONNECT_TIMEOUT and REQUEST_TIMEOUT and the concurrency setting against the pool size. Done means the maintainers have agreed on the lease-based claim design or the startup guard, with tests covering a delivery that outlives the server's idle timeout.

由索引模型根據 Issue 內容生成。

描述

enhancement

Severity: Medium-High (pool starvation, duplicate-delivery loop, DB bloat) · Area: processor transaction design · Found in: code review of main @ e5630a1

Problem

The whole cycle (claim with FOR UPDATE SKIP LOCKED → network delivery → status update) runs in one DB transaction, so a pooled connection sits idle in transaction for as long as the transport I/O takes:

  • HTTP: up to 10 s connect + 30 s request timeout (hard-coded CONNECT_TIMEOUT/REQUEST_TIMEOUT, not configurable)
  • Kafka: producer.flush() blocks up to delivery.timeout.ms (120 s default)

Concrete failure modes:

  1. Connection-pool starvation. concurrency accepts values up to 256, and each worker holds one connection for the full delivery. Nothing checks this against the pool size: Hikari's default is 10. With concurrency >= pool size and a slow downstream, okapi takes every connection and the application's own requests time out on getConnection().
  2. Infinite duplicate loop under server-side timeouts. With Postgres idle_in_transaction_session_timeout, or a MySQL wait_timeout/proxy idle timeout shorter than the delivery time, the server kills the session mid-delivery. The final update/commit fails, and every entry that was actually delivered reverts to PENDING with retries unchanged. It is delivered again on the next tick, killed again, and so on for as long as the downstream stays slow. retries never advances, so nothing ever reaches FAILED.
  3. MVCC bloat (Postgres). Long transactions hold back the xmin horizon on what is by design a very high-churn table (every row is inserted, then updated, then deleted), which degrades vacuum for the entire database.
  4. Combined with the MySQL gap-lock issue, publishers are blocked for the same duration.

The KDoc on CompositeMessageDeliverer acknowledges the open transaction, but the consequences above aren't documented or guarded against.

Suggested fix

  • Preferred: a lease-based claim. In a short transaction, mark rows PROCESSING with locked_until = now + leaseTimeout (or bump a claim_token), commit, deliver outside any transaction, then write results in a second short transaction guarded by the claim token. A reaper (or the claim query itself) reclaims rows whose lease has expired.
  • At minimum: fail fast or warn at startup when concurrency ≥ the pool's maximum size; make the HTTP timeouts configurable; document the required relationship between delivery timeouts and idle_in_transaction_session_timeout/lock-wait timeouts.
主要語言
Kotlin
星號
14
分支
0
平均合併
14 小時 16 分鐘
30 天內合併 PR
9

環境準備

這個專案沒有提供開發容器、Dockerfile 或貢獻指南，環境需要你自己搭建：先看它的 README，通用步驟見我們的新手貢獻指南。

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

softwaremill/okapi 的其他 Issue

查看 softwaremill/okapi 的全部 Issue

相似的 Issue

更多 Kotlin Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。