3.2.2: proxy freezes with 100% CPU after an upstream broker restart (persistent client connections)

Aperta
#259 2 commenti 1 reazione 1 assegnatario Vedi su GitHub

@carlhoerberg ci sta già lavorando.

Dal 13/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Since 3.2.2, restarting the upstream RabbitMQ broker while clients keep persistent connections to amqproxy can leave the proxy stuck: one CPU at 100%, no more messages relayed, and /healthz not responding until the container is restarted. The AMQP port still accepts TCP connections in that state.

Environment: amqproxy 3.2.2 (official image) as a Kubernetes sidecar, upstream RabbitMQ 3.13 behind a ClusterIP Service, PHP ext-amqp 2.2.0 clients using persistent connections (pconnect), heartbeat 0.

Reproduction: publish one message every 200 ms through the proxy on a persistent connection, then delete the broker pod. The freeze starts right after the log line "No Connection#CloseOk ... closing connection". strace shows thousands of connect() calls and no epoll_wait, so one fiber appears never to yield.

Frequency: 1 out of 5 runs on a local kind cluster, 2 out of 2 in our integration cluster. Never seen with 3.2.1 (0/5) or 3.1.3. A 3.2.2 build with Crystal 1.17 also froze (1/5), which points to the change in #256 (wait_for_close_ok) rather than to the Crystal 1.21 upgrade.

Related: with non-persistent clients, the proxy can also stall for about 2 minutes while the broker is down. This is the upstream dial without timeout under the pool lock described in #258.

Happy to provide logs or test a patch.

Lingua principale
Crystal
Stelle
408
Fork
50
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di cloudamqp/amqproxy

Tutte le issue di cloudamqp/amqproxy

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.