Delivery I/O runs inside the claim transaction: pool starvation, idle-in-transaction kills cause infinite redelivery, MVCC bloat

Offen
  • Stimmt etwas nicht?
#133 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
22/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
kafka, kotlin, mysql, postgresql
Bereich
backend, databases, distributed-systems

Rechercherichtung

Start with the KDoc on CompositeMessageDeliverer, which acknowledges the open transaction, and trace where the claim, delivery and status update share one transaction. Check the hard-coded CONNECT_TIMEOUT and REQUEST_TIMEOUT and the concurrency setting against the pool size. Done means the maintainers have agreed on the lease-based claim design or the startup guard, with tests covering a delivery that outlives the server's idle timeout.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

enhancement

Severity: Medium-High (pool starvation, duplicate-delivery loop, DB bloat) · Area: processor transaction design · Found in: code review of main @ e5630a1

Problem

The whole cycle (claim with FOR UPDATE SKIP LOCKED → network delivery → status update) runs in one DB transaction, so a pooled connection sits idle in transaction for as long as the transport I/O takes:

  • HTTP: up to 10 s connect + 30 s request timeout (hard-coded CONNECT_TIMEOUT/REQUEST_TIMEOUT, not configurable)
  • Kafka: producer.flush() blocks up to delivery.timeout.ms (120 s default)

Concrete failure modes:

  1. Connection-pool starvation. concurrency accepts values up to 256, and each worker holds one connection for the full delivery. Nothing checks this against the pool size: Hikari's default is 10. With concurrency >= pool size and a slow downstream, okapi takes every connection and the application's own requests time out on getConnection().
  2. Infinite duplicate loop under server-side timeouts. With Postgres idle_in_transaction_session_timeout, or a MySQL wait_timeout/proxy idle timeout shorter than the delivery time, the server kills the session mid-delivery. The final update/commit fails, and every entry that was actually delivered reverts to PENDING with retries unchanged. It is delivered again on the next tick, killed again, and so on for as long as the downstream stays slow. retries never advances, so nothing ever reaches FAILED.
  3. MVCC bloat (Postgres). Long transactions hold back the xmin horizon on what is by design a very high-churn table (every row is inserted, then updated, then deleted), which degrades vacuum for the entire database.
  4. Combined with the MySQL gap-lock issue, publishers are blocked for the same duration.

The KDoc on CompositeMessageDeliverer acknowledges the open transaction, but the consequences above aren't documented or guarded against.

Suggested fix

  • Preferred: a lease-based claim. In a short transaction, mark rows PROCESSING with locked_until = now + leaseTimeout (or bump a claim_token), commit, deliver outside any transaction, then write results in a second short transaction guarded by the claim token. A reaper (or the claim query itself) reclaims rows whose lease has expired.
  • At minimum: fail fast or warn at startup when concurrency ≥ the pool's maximum size; make the HTTP timeouts configurable; document the required relationship between delivery timeouts and idle_in_transaction_session_timeout/lock-wait timeouts.
Vorherrschende Sprache
Kotlin
Sterne
14
Forks
0
Ø Merge
9 Std. 54 Min.
Gemergte PRs (30 T.)
11

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus softwaremill/okapi

Alle Issues in softwaremill/okapi

Ähnliche Issues

Weitere Issues zu Kotlin

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.