Duplicate Pub/Sub messages after Redis Sentinel failover

Abierto
  • ¿Algo no está bien?
#3,336 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@vkyow ya está trabajando en esto.

Desde el 6/11/2026.

  • #3337 de @vkyow — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
csharp, redis
Área
backend, databases

Línea de trabajo

Read Subscription.cs, ResultProcessor.cs, and ServerEndPoint.cs at the cited commit, then inspect the attached regression test and run its eight cases against the current source. The issue reports those cases failing and links an open pull request for a proposed fix; completion means preventing duplicate deliveries after failover while retaining the stated message loss behavior during failover.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Duplicate Pub/Sub messages after Redis Sentinel failover

I'm seeing an intermittent issue where the same Pub/Sub message is delivered twice after a Sentinel failover. It doesn't happen on every switch, but when it does, the duplicates continue after the connection has recovered. In one test they continued for the full two minutes I kept traffic running.

I reproduced this in a standalone .NET console application using StackExchange.Redis directly. It subscribes once to a single pattern, and a separate publisher sends each numbered message once.

Environment

  • StackExchange.Redis 3.4.31 from NuGet.
  • Redis and Sentinel 8.0.3.
  • .NET 10, running in Linux Docker containers on Docker Desktop for Windows.
  • Tested with both RESP2 and RESP3. Either protocol can reproduce it, although the result varies between runs.
  • Two setups: one primary + two replicas with three Sentinels, and one primary + four replicas with five Sentinels. Quorum was two and three respectively.

How to reproduce

  1. Connect through Sentinel using serviceName=mymaster and subscribe once to repro.events.*.

  2. From a separate publisher, send a unique numbered ID to repro.events.0 about every 100 ms. Send each ID once, directly to the primary reported by Sentinel.

  3. Keep publishing and trigger a failover:

    redis-cli -h sentinel-0 -p 26379 SENTINEL FAILOVER mymaster

  4. Leave the consumer running and check how many times each ID reaches the callback. Repeat the switch if necessary, since the issue is intermittent.

The consumer is essentially:

using var connection = await ConnectionMultiplexer.ConnectAsync(
    "sentinel-0:26379,sentinel-1:26379,sentinel-2:26379," +
    "serviceName=mymaster,abortConnect=false," +
    "connectTimeout=5000,syncTimeout=5000,protocol=resp3");

var counts = new ConcurrentDictionary<string, int>();
await connection.GetSubscriber().SubscribeAsync(
    RedisChannel.Pattern("repro.events.*"), (_, message) =>
    {
        var id = message.ToString();
        var count = counts.AddOrUpdate(id, 1, (_, previous) => previous + 1);
        Console.WriteLine($"id={id} count={count}");
    });

There is only one subscription, with no overlapping literal subscription or extra calls to Subscribe on reconnect. The attached project contains the complete subscriber example. It needs a reachable Sentinel setup and a separate publisher.

What I observed

In one five-node test, the publisher sent 1,310 unique IDs. The RESP3 consumer received 1,226 of those IDs more than once. Packet captures showed one PUBLISH per ID, followed by two pmessage deliveries from different Redis nodes to the same consumer.

For example:

Message ID:           promotion1-85
PUBLISH commands:     1
First delivery from:  172.20.0.5:6379
Second delivery from: 172.20.0.3:6379

The consumer still had subscriptions on two Redis nodes after failover had settled. RESP2 recovered normally in that particular run. In a separate three-node test, both protocols duplicated messages: 1,226 repeated IDs with RESP2 and 1,184 with RESP3.

I expect some messages to be lost while failover is happening. The concern here is the persistent duplicate delivery once traffic has resumed. The captures used for these examples had no reported packet drops or parsing gaps.

Possible cause

Looking at the source, I suspect two subscription-restoration paths can overlap before the first SUBSCRIBE/PSUBSCRIBE acknowledgement arrives:

  • ServerEndPoint.OnFullyEstablished and ConnectionMultiplexer.ReconfigureAsync can both call EnsureSubscriptions(CommandFlags.FireAndForget).
  • SingleNodeSubscription records its current server when the acknowledgement is processed. Until then, another restore can still see the subscription as disconnected and choose another node.
  • If both attempts succeed, the later acknowledgement replaces the recorded endpoint while the other physical subscription remains active.

This would explain the two deliveries from different nodes. One of those nodes can be a replica; it isn't always the old primary.

The source I checked was commit 9d17974fcade6c89b29b233030fe281e75f9ab1c, particularly Subscription.cs, ResultProcessor.cs, and ServerEndPoint.cs.

Regression test and local patch

I've attached a test using the repository's in-process test server. It holds the first subscription response, starts a restore to a second node, and checks whether a second registration is created before the first is acknowledged. The initial eight cases failed against the unchanged source.

I also have a local patch that coordinates restoration through the acknowledgement and prevents a late response from replacing a newer subscription. It also cleans up obsolete registrations and checks the source connection before dispatching a message.

The patched client passed 104 recovery events across the three-node and five-node setups, with no duplicate frames or callbacks observed. The five-node tests included Sentinel restarts and an outage with the primary and two Sentinels paused. Messages were still lost during failover, so this doesn't provide replay or a general exactly-once guarantee.

I also noticed recovery delays from concurrent Sentinel discovery queries. The local candidate includes an additional change to acquire the reconfiguration gate before discovery; that could be reviewed separately.

issue-attachments.zip

Lenguaje dominante
C#
Estrellas
6.2k
Forks
1.6k
Merge medio
9 h 42 min
PR fusionados (30 d)
51

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de StackExchange/StackExchange.Redis

Todos los issues de StackExchange/StackExchange.Redis

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.