jondot/sneakers

When RabbitMQ node is unreachable, workers try to re-connect and subscribe without a delay between retries

Aperta

#290 aperta il 24 mag 2017

Vedi su GitHub
 (4 commenti) (2 reazioni) (0 assegnatari)Ruby (319 fork)github user discovery
bughelp wanted

Metriche repository

Star
 (2239 stelle)
Metriche merge PR
 (Nessuna PR mergiata in 30 g)

Descrizione

I have a Sneakers process configured to spawn 4 workers, which I have set up to run on deploy using the capistrano-sneakers gem. Normally I will see this on a fresh deploy:

2017-05-24T02:32:05Z p-13632 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:jy4cit][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed838>] New worker: subscribing.
2017-05-24T02:32:05Z p-13632 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:jy4cit][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed838>] New worker: I'm alive.
2017-05-24T02:32:05Z p-13721 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:popuet][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed478>] New worker: subscribing.
2017-05-24T02:32:05Z p-13736 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:ef4n31][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed0e0>] New worker: subscribing.
2017-05-24T02:32:05Z p-13721 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:popuet][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed478>] New worker: I'm alive.
2017-05-24T02:32:05Z p-13736 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:ef4n31][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed0e0>] New worker: I'm alive.
2017-05-24T02:32:06Z p-13921 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:s8mlwf][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aecd48>] New worker: subscribing.
2017-05-24T02:32:06Z p-13921 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:s8mlwf][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aecd48>] New worker: I'm alive.

However, occasionally I will just get a stream of 'New worker: subscribing' messages:

2017-05-24T02:31:49Z p-11498 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:fzkcmj][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8de60>] New worker: subscribing.
2017-05-24T02:31:49Z p-11513 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:8c21au][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8d938>] New worker: subscribing.
2017-05-24T02:31:49Z p-11528 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:1ft0od][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8d3c0>] New worker: subscribing.
2017-05-24T02:31:49Z p-11609 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:615obl][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8ce48>] New worker: subscribing.
2017-05-24T02:31:49Z p-11707 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:zpc804][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8c358>] New worker: subscribing.
2017-05-24T02:31:49Z p-11782 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:8dowvs][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbfd98>] New worker: subscribing.
2017-05-24T02:31:49Z p-11868 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:d6cv6d][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbf578>] New worker: subscribing.
2017-05-24T02:31:49Z p-11989 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:pjxo7s][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbef88>] New worker: subscribing.
2017-05-24T02:31:49Z p-12019 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:jzllpi][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbe290>] New worker: subscribing.
2017-05-24T02:31:50Z p-12036 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:qksstj][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbdcf0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12051 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:yu57p3][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbd610>] New worker: subscribing.
2017-05-24T02:31:50Z p-12066 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:xrztgd][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbcf58>] New worker: subscribing.
2017-05-24T02:31:50Z p-12084 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:yervvu][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbc288>] New worker: subscribing.
2017-05-24T02:31:50Z p-12167 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:on1mtd][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9bce0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12263 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:v9sy8n][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9b768>] New worker: subscribing.
2017-05-24T02:31:50Z p-12352 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:mj30hc][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9b1f0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12425 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:4pn9do][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9a700>] New worker: subscribing.
2017-05-24T02:31:50Z p-12513 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:5ihvog][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9a188>] New worker: subscribing.
2017-05-24T02:31:50Z p-12602 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:wn03ek][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae99c10>] New worker: subscribing.

~500 a minute, until I stop the process or do another deploy. It's intermittent, I can do two consecutive deploys with no code changes and it will happen in one and not the other.

Guida contributor

Direzione di ricerca
Esaminare la logica di riconnessione nel codice del worker di Sneakers, in particolare il metodo `subscribe` e le impostazioni di riconnessione del client Bunny. Verificare se c'è un ritardo tra i tentativi di riconnessione quando la connessione fallisce, e considerare l'aggiunta di un backoff esponenziale o di un intervallo di retry configurabile.
Tech stack
ruby
Dominio
backend
Tipo issue
Bug
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
Mezza giornata
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
Attiva
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
Chiara
Prerequisiti
RubyRabbitMQ basics
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
60