Delivery I/O runs inside the claim transaction: pool starvation, idle-in-transaction kills cause infinite redelivery, MVCC bloat

Ouverte
  • Quelque chose ne va pas ?
#133 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
22/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
kafka, kotlin, mysql, postgresql
Domaine
backend, databases, distributed-systems

Piste de recherche

Start with the KDoc on CompositeMessageDeliverer, which acknowledges the open transaction, and trace where the claim, delivery and status update share one transaction. Check the hard-coded CONNECT_TIMEOUT and REQUEST_TIMEOUT and the concurrency setting against the pool size. Done means the maintainers have agreed on the lease-based claim design or the startup guard, with tests covering a delivery that outlives the server's idle timeout.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement

Severity: Medium-High (pool starvation, duplicate-delivery loop, DB bloat) · Area: processor transaction design · Found in: code review of main @ e5630a1

Problem

The whole cycle (claim with FOR UPDATE SKIP LOCKED → network delivery → status update) runs in one DB transaction, so a pooled connection sits idle in transaction for as long as the transport I/O takes:

  • HTTP: up to 10 s connect + 30 s request timeout (hard-coded CONNECT_TIMEOUT/REQUEST_TIMEOUT, not configurable)
  • Kafka: producer.flush() blocks up to delivery.timeout.ms (120 s default)

Concrete failure modes:

  1. Connection-pool starvation. concurrency accepts values up to 256, and each worker holds one connection for the full delivery. Nothing checks this against the pool size: Hikari's default is 10. With concurrency >= pool size and a slow downstream, okapi takes every connection and the application's own requests time out on getConnection().
  2. Infinite duplicate loop under server-side timeouts. With Postgres idle_in_transaction_session_timeout, or a MySQL wait_timeout/proxy idle timeout shorter than the delivery time, the server kills the session mid-delivery. The final update/commit fails, and every entry that was actually delivered reverts to PENDING with retries unchanged. It is delivered again on the next tick, killed again, and so on for as long as the downstream stays slow. retries never advances, so nothing ever reaches FAILED.
  3. MVCC bloat (Postgres). Long transactions hold back the xmin horizon on what is by design a very high-churn table (every row is inserted, then updated, then deleted), which degrades vacuum for the entire database.
  4. Combined with the MySQL gap-lock issue, publishers are blocked for the same duration.

The KDoc on CompositeMessageDeliverer acknowledges the open transaction, but the consequences above aren't documented or guarded against.

Suggested fix

  • Preferred: a lease-based claim. In a short transaction, mark rows PROCESSING with locked_until = now + leaseTimeout (or bump a claim_token), commit, deliver outside any transaction, then write results in a second short transaction guarded by the claim token. A reaper (or the claim query itself) reclaims rows whose lease has expired.
  • At minimum: fail fast or warn at startup when concurrency ≥ the pool's maximum size; make the HTTP timeouts configurable; document the required relationship between delivery timeouts and idle_in_transaction_session_timeout/lock-wait timeouts.
Langage dominant
Kotlin
Étoiles
14
Forks
0
Merge moyen
9 h 54 min
PR mergées (30 j)
11

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de softwaremill/okapi

Toutes les issues de softwaremill/okapi

Issues similaires

Plus d'issues Kotlin

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.