jondot/sneakers

When RabbitMQ node is unreachable, workers try to re-connect and subscribe without a delay between retries

Open

#290 opened on May 24, 2017

View on GitHub
 (4 comments) (2 reactions) (0 assignees)Ruby (319 forks)github user discovery
bughelp wanted

Repository metrics

Stars
 (2,239 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

I have a Sneakers process configured to spawn 4 workers, which I have set up to run on deploy using the capistrano-sneakers gem. Normally I will see this on a fresh deploy:

2017-05-24T02:32:05Z p-13632 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:jy4cit][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed838>] New worker: subscribing.
2017-05-24T02:32:05Z p-13632 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:jy4cit][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed838>] New worker: I'm alive.
2017-05-24T02:32:05Z p-13721 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:popuet][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed478>] New worker: subscribing.
2017-05-24T02:32:05Z p-13736 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:ef4n31][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed0e0>] New worker: subscribing.
2017-05-24T02:32:05Z p-13721 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:popuet][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed478>] New worker: I'm alive.
2017-05-24T02:32:05Z p-13736 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:ef4n31][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aed0e0>] New worker: I'm alive.
2017-05-24T02:32:06Z p-13921 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:s8mlwf][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aecd48>] New worker: subscribing.
2017-05-24T02:32:06Z p-13921 t-6fosc DEBUG: [worker-inbound_subscriber_sandbox:1:s8mlwf][#<Thread:0x0000000149e278>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009aecd48>] New worker: I'm alive.

However, occasionally I will just get a stream of 'New worker: subscribing' messages:

2017-05-24T02:31:49Z p-11498 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:fzkcmj][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8de60>] New worker: subscribing.
2017-05-24T02:31:49Z p-11513 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:8c21au][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8d938>] New worker: subscribing.
2017-05-24T02:31:49Z p-11528 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:1ft0od][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8d3c0>] New worker: subscribing.
2017-05-24T02:31:49Z p-11609 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:615obl][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8ce48>] New worker: subscribing.
2017-05-24T02:31:49Z p-11707 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:zpc804][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae8c358>] New worker: subscribing.
2017-05-24T02:31:49Z p-11782 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:8dowvs][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbfd98>] New worker: subscribing.
2017-05-24T02:31:49Z p-11868 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:d6cv6d][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbf578>] New worker: subscribing.
2017-05-24T02:31:49Z p-11989 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:pjxo7s][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbef88>] New worker: subscribing.
2017-05-24T02:31:49Z p-12019 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:jzllpi][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbe290>] New worker: subscribing.
2017-05-24T02:31:50Z p-12036 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:qksstj][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbdcf0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12051 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:yu57p3][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbd610>] New worker: subscribing.
2017-05-24T02:31:50Z p-12066 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:xrztgd][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbcf58>] New worker: subscribing.
2017-05-24T02:31:50Z p-12084 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:yervvu][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x00000009fbc288>] New worker: subscribing.
2017-05-24T02:31:50Z p-12167 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:on1mtd][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9bce0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12263 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:v9sy8n][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9b768>] New worker: subscribing.
2017-05-24T02:31:50Z p-12352 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:mj30hc][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9b1f0>] New worker: subscribing.
2017-05-24T02:31:50Z p-12425 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:4pn9do][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9a700>] New worker: subscribing.
2017-05-24T02:31:50Z p-12513 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:5ihvog][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae9a188>] New worker: subscribing.
2017-05-24T02:31:50Z p-12602 t-bddn0 DEBUG: [worker-inbound_subscriber_sandbox:1:wn03ek][#<Thread:0x0000000246e298>][inbound_subscriber_sandbox][#<Sneakers::Configuration:0x0000000ae99c10>] New worker: subscribing.

~500 a minute, until I stop the process or do another deploy. It's intermittent, I can do two consecutive deploys with no code changes and it will happen in one and not the other.

Contributor guide

Research direction
Look at the reconnection logic in Sneakers' worker code, specifically the `subscribe` method and the Bunny client's reconnection settings. Check if there is a delay between retry attempts when the connection fails, and consider adding an exponential backoff or a configurable retry interval.
Tech stack
ruby
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
RubyRabbitMQ basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60