More aggressively retry Redis commands
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- redis, ruby
- Domain
- backend, distributed-systems
Research direction
No files or tests are named. Start by inventorying every Redis command emitted by ci-queue and reviewing the redis gem's retry configuration; done means transient Redis failures are retried without allowing non-idempotent commands to corrupt state.
Written by the indexing model from the issue text.
Description
Acceptance Criteria
- We need to go over all the commands we emit and make sure they are idempotent, otherwise retrying could result in a corrupted state, lost tests etc.
- The redis gem has the necessary elements for that it's mostly just configuration.
Context
Sometimes our Redis server that handle the ci-queue workload experience a failover or some other availability issues.
When this happens it break builds even though it recovers pretty fast.
Examples
Error connecting to Redis on redacted.svc.cluster.local.:6379 (SocketError) (Redis::CannotConnectError)
./tmp/bundle/ruby/3.1.0/gems/redis-4.8.0/lib/redis/client.rb:162:in `call': MASTERDOWN Link with MASTER is down and replica-serve-stale-data is set to 'no'. (Redis::CommandError)
(that later one need to be better categorized by the redis gem though)
Solution
Ideally we'd be resilient to these small transient errors, this means retrying all or most commands and possibly waiting a bit before retrying. The redis gem has the necessary elements for that it's mostly just configuration.
However we need to go over all the commands we emit and make sure they are idempotent, otherwise retrying could result in a corrupted state, lost tests etc.
cc @ChrisBr
- Dominant language
- Ruby
- Stars
- 240
- Forks
- 36
- Avg merge
- 28m
- Merged PRs (30d)
- 3
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Shopify/ci-queue
-
Difficulty 3/5 1-2 days Newbie friendliness 65/100
-
Difficulty 4/5 3-5 days Newbie friendliness 50/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 45/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 38/100
All issues in Shopify/ci-queue
Similar issues
-
SyncEm always forwards a dummy block, so wrapped methods lose their no-block/Enumerator behavior Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
endoflife-date/endoflife.date#11086 ·
-
internal
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Kong/developer.konghq.com#7322 ·
-
bug P2
Difficulty 2/5 1-3 hours Newbie friendliness 88/100