[Bug] Unrecoverable deadlock when an in-progress config cannot complete

Open
  • Is something wrong?
#42 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
devops, infrastructure

Research direction

Start by tracing the RedkeyConfig reconciliation flow, especially how InProgress, Applied, and Pending configurations are sequenced. Compare the timeout, rollback, and forced/priority options, then define the recovery behavior and add unit/integration coverage for an invalid image blocking an upgrade followed by corrective recovery.

Written by the indexing model from the issue text.

Description

bug

Summary

A stuck RedkeyConfig in InProgress blocks every subsequent configuration change, leaving the cluster with no way to recover.

Context

The Operator creates sequenced RedkeyConfig CRs and Robin applies them one at a time, in sequence order. A config in InProgress keeps being reconciled until it reaches Applied; only then does Robin move on to the next Pending config.

Problem

If a change makes the in-progress config impossible to complete, Robin can never advance:

  • Example: changing the Redis image to a tag that does not exist. The new pods fail to start (Crash/ImagePullBackOff) and the config never reaches Applied.
  • Because the in-progress config never completes, all later Pending configs are blocked.
  • Applying a corrective change afterwards does not help: the new config sits behind the stuck one and is never picked up.
  • There is currently no mechanism to unblock the cluster.

Impact

A single bad configuration change wedges the cluster indefinitely, and the operator cannot self-heal or accept a fix.

Options considered

  1. Timeout → Error status. Move the config to an Error state if it does not complete within a timeout. Not preferred: it is very hard to pick a single timeout that fits all operations and all clusters.
  2. Rollback mechanism. Discard the in-progress operation and revert to the last successfully applied configuration. Potentially complex: depending on the state the cluster was left in, returning to the previous configuration may not be trivial.
  3. Forced / priority configuration. Allow marking a new config as "forced" so Robin prioritizes it, sets aside the in-progress one, and applies the new configuration instead.

Acceptance criteria

  • Decide and document a recovery strategy (option 1, 2, 3, or a combination).
  • A user can unblock a wedged cluster and apply a corrective configuration without manual CR surgery.
  • Behavior is covered by unit/integration tests, including the "invalid image blocks upgrade, then a corrective config recovers it" scenario.

Related

  • Partially mitigated by #72 for the specific case where the desired end state is a scale-to-zero.
Dominant language
Go
Stars
18
Forks
0
Avg merge
11h 16m
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from InditexTech/redkey-robin

All issues in InditexTech/redkey-robin

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.