Duplicate Pub/Sub messages after Redis Sentinel failover

Offen
  • Stimmt etwas nicht?
#3,336 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@vkyow arbeitet bereits daran.

Seit 06.11.2026.

  • #3337 von @vkyow — offen

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
25/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
csharp, redis
Bereich
backend, databases

Rechercherichtung

Read Subscription.cs, ResultProcessor.cs, and ServerEndPoint.cs at the cited commit, then inspect the attached regression test and run its eight cases against the current source. The issue reports those cases failing and links an open pull request for a proposed fix; completion means preventing duplicate deliveries after failover while retaining the stated message loss behavior during failover.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Duplicate Pub/Sub messages after Redis Sentinel failover

I'm seeing an intermittent issue where the same Pub/Sub message is delivered twice after a Sentinel failover. It doesn't happen on every switch, but when it does, the duplicates continue after the connection has recovered. In one test they continued for the full two minutes I kept traffic running.

I reproduced this in a standalone .NET console application using StackExchange.Redis directly. It subscribes once to a single pattern, and a separate publisher sends each numbered message once.

Environment

  • StackExchange.Redis 3.4.31 from NuGet.
  • Redis and Sentinel 8.0.3.
  • .NET 10, running in Linux Docker containers on Docker Desktop for Windows.
  • Tested with both RESP2 and RESP3. Either protocol can reproduce it, although the result varies between runs.
  • Two setups: one primary + two replicas with three Sentinels, and one primary + four replicas with five Sentinels. Quorum was two and three respectively.

How to reproduce

  1. Connect through Sentinel using serviceName=mymaster and subscribe once to repro.events.*.

  2. From a separate publisher, send a unique numbered ID to repro.events.0 about every 100 ms. Send each ID once, directly to the primary reported by Sentinel.

  3. Keep publishing and trigger a failover:

    redis-cli -h sentinel-0 -p 26379 SENTINEL FAILOVER mymaster

  4. Leave the consumer running and check how many times each ID reaches the callback. Repeat the switch if necessary, since the issue is intermittent.

The consumer is essentially:

using var connection = await ConnectionMultiplexer.ConnectAsync(
    "sentinel-0:26379,sentinel-1:26379,sentinel-2:26379," +
    "serviceName=mymaster,abortConnect=false," +
    "connectTimeout=5000,syncTimeout=5000,protocol=resp3");

var counts = new ConcurrentDictionary<string, int>();
await connection.GetSubscriber().SubscribeAsync(
    RedisChannel.Pattern("repro.events.*"), (_, message) =>
    {
        var id = message.ToString();
        var count = counts.AddOrUpdate(id, 1, (_, previous) => previous + 1);
        Console.WriteLine($"id={id} count={count}");
    });

There is only one subscription, with no overlapping literal subscription or extra calls to Subscribe on reconnect. The attached project contains the complete subscriber example. It needs a reachable Sentinel setup and a separate publisher.

What I observed

In one five-node test, the publisher sent 1,310 unique IDs. The RESP3 consumer received 1,226 of those IDs more than once. Packet captures showed one PUBLISH per ID, followed by two pmessage deliveries from different Redis nodes to the same consumer.

For example:

Message ID:           promotion1-85
PUBLISH commands:     1
First delivery from:  172.20.0.5:6379
Second delivery from: 172.20.0.3:6379

The consumer still had subscriptions on two Redis nodes after failover had settled. RESP2 recovered normally in that particular run. In a separate three-node test, both protocols duplicated messages: 1,226 repeated IDs with RESP2 and 1,184 with RESP3.

I expect some messages to be lost while failover is happening. The concern here is the persistent duplicate delivery once traffic has resumed. The captures used for these examples had no reported packet drops or parsing gaps.

Possible cause

Looking at the source, I suspect two subscription-restoration paths can overlap before the first SUBSCRIBE/PSUBSCRIBE acknowledgement arrives:

  • ServerEndPoint.OnFullyEstablished and ConnectionMultiplexer.ReconfigureAsync can both call EnsureSubscriptions(CommandFlags.FireAndForget).
  • SingleNodeSubscription records its current server when the acknowledgement is processed. Until then, another restore can still see the subscription as disconnected and choose another node.
  • If both attempts succeed, the later acknowledgement replaces the recorded endpoint while the other physical subscription remains active.

This would explain the two deliveries from different nodes. One of those nodes can be a replica; it isn't always the old primary.

The source I checked was commit 9d17974fcade6c89b29b233030fe281e75f9ab1c, particularly Subscription.cs, ResultProcessor.cs, and ServerEndPoint.cs.

Regression test and local patch

I've attached a test using the repository's in-process test server. It holds the first subscription response, starts a restore to a second node, and checks whether a second registration is created before the first is acknowledged. The initial eight cases failed against the unchanged source.

I also have a local patch that coordinates restoration through the acknowledgement and prevents a late response from replacing a newer subscription. It also cleans up obsolete registrations and checks the source connection before dispatching a message.

The patched client passed 104 recovery events across the three-node and five-node setups, with no duplicate frames or callbacks observed. The five-node tests included Sentinel restarts and an outage with the primary and two Sentinels paused. Messages were still lost during failover, so this doesn't provide replay or a general exactly-once guarantee.

I also noticed recovery delays from concurrent Sentinel discovery queries. The local candidate includes an additional change to acquire the reconfiguration gate before discovery; that could be reviewed separately.

issue-attachments.zip

Vorherrschende Sprache
C#
Sterne
6.2k
Forks
1.6k
Ø Merge
9 Std. 42 Min.
Gemergte PRs (30 T.)
51

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus StackExchange/StackExchange.Redis

Alle Issues in StackExchange/StackExchange.Redis

Ähnliche Issues

Weitere Issues zu C#

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.