OT-CONTAINER-KIT/redis-operator

Data loss during cluster fail over.

Open

#1,158 opened on Dec 10, 2024

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Go (377 forks)auto 404
bughelp wanted

Repository metrics

Stars
 (1,416 stars)
PR merge metrics
 (PR metrics pending)

Description

What version of redis operator are you using?

redis-operator version: ghcr.io/ot-container-kit/redis-operator/redis-operator:v0.18.1

Does this issue reproduce with the latest release? yes

What operating system and processor architecture are you using (kubectl version)?

What did you do?

Created sentinel and replication cluster with below configuration. redis-replication clusterSize = 2 redis-sentinel clusterSize = 3

Node1 contains redis-replication-0, sentinel-0 and sentinel-2 instances. Node2 contains redis-replication-1 and sentinel-1 instances

Disabled scheduling on Node1. Manually deleted master instance (redis-replication-0). Sentinel elected new master (redis-replication-1) and able to write data. Enabled scheduling on Node1, when old master instance(redis-replication-0) comes up, data written into master (redis-replication-1) is not present. Basically data written to master (redis-replication-1) is replaced with old master(redis-replication-0) data.

What did you expect to see?

Data should persist in newly elected master.

What did you see instead?

Data loss in newly elected master.

Contributor guide

Research direction
Investigate the failover logic in the redis operator, specifically how nodes rejoin the cluster after being rescheduled. Check if the old master is reconfigured as a replica and whether the data sync happens correctly. Look at the Sentinel configuration and the reconciliation loop.
Tech stack
go
Domain
databasebackend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GoKubernetesRedis
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40