Delivery I/O runs inside the claim transaction: pool starvation, idle-in-transaction kills cause infinite redelivery, MVCC bloat

Aperta
  • Qualcosa non va?
#133 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
22/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
kafka, kotlin, mysql, postgresql
Ambito
backend, databases, distributed-systems

Direzione di ricerca

Start with the KDoc on CompositeMessageDeliverer, which acknowledges the open transaction, and trace where the claim, delivery and status update share one transaction. Check the hard-coded CONNECT_TIMEOUT and REQUEST_TIMEOUT and the concurrency setting against the pool size. Done means the maintainers have agreed on the lease-based claim design or the startup guard, with tests covering a delivery that outlives the server's idle timeout.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Severity: Medium-High (pool starvation, duplicate-delivery loop, DB bloat) · Area: processor transaction design · Found in: code review of main @ e5630a1

Problem

The whole cycle (claim with FOR UPDATE SKIP LOCKED → network delivery → status update) runs in one DB transaction, so a pooled connection sits idle in transaction for as long as the transport I/O takes:

  • HTTP: up to 10 s connect + 30 s request timeout (hard-coded CONNECT_TIMEOUT/REQUEST_TIMEOUT, not configurable)
  • Kafka: producer.flush() blocks up to delivery.timeout.ms (120 s default)

Concrete failure modes:

  1. Connection-pool starvation. concurrency accepts values up to 256, and each worker holds one connection for the full delivery. Nothing checks this against the pool size: Hikari's default is 10. With concurrency >= pool size and a slow downstream, okapi takes every connection and the application's own requests time out on getConnection().
  2. Infinite duplicate loop under server-side timeouts. With Postgres idle_in_transaction_session_timeout, or a MySQL wait_timeout/proxy idle timeout shorter than the delivery time, the server kills the session mid-delivery. The final update/commit fails, and every entry that was actually delivered reverts to PENDING with retries unchanged. It is delivered again on the next tick, killed again, and so on for as long as the downstream stays slow. retries never advances, so nothing ever reaches FAILED.
  3. MVCC bloat (Postgres). Long transactions hold back the xmin horizon on what is by design a very high-churn table (every row is inserted, then updated, then deleted), which degrades vacuum for the entire database.
  4. Combined with the MySQL gap-lock issue, publishers are blocked for the same duration.

The KDoc on CompositeMessageDeliverer acknowledges the open transaction, but the consequences above aren't documented or guarded against.

Suggested fix

  • Preferred: a lease-based claim. In a short transaction, mark rows PROCESSING with locked_until = now + leaseTimeout (or bump a claim_token), commit, deliver outside any transaction, then write results in a second short transaction guarded by the claim token. A reaper (or the claim query itself) reclaims rows whose lease has expired.
  • At minimum: fail fast or warn at startup when concurrency ≥ the pool's maximum size; make the HTTP timeouts configurable; document the required relationship between delivery timeouts and idle_in_transaction_session_timeout/lock-wait timeouts.
Lingua principale
Kotlin
Stelle
14
Fork
0
Merge medio
9h 54m
PR unite (30g)
11

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di softwaremill/okapi

Tutte le issue di softwaremill/okapi

Issue simili

Altre issue su Kotlin

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.