Coordinator singleton lease loss permanently stops the local ShardRegion while the node is still Up

Read ShardCoordinator.scala terminate() (lines 964-978) and ClusterSharding.scala regions cache (line 189). Follow reproduction recipe in ClusterShardingLeaseSpec with coordinator-singleton.use-lease set; step 4 asserting shardRegion still delivers should fail pre-fix. Done: local ShardRegion survives coordinator lease loss and messages route normally.

Summary

When the shard coordinator singleton's lease is lost, ClusterSingletonManager sends the configured terminationMessage to the singleton and returns to AcquiringLease, intending to re-acquire and carry on. For cluster sharding that message is ShardCoordinator.Internal.Terminate, which makes the coordinator gracefully shut down the local ShardRegion and then stop itself. The node is still Up and still supposed to host shards, but its local region is gone and is never recreated, because ClusterSharding caches the region ref and start short-circuits on the cache.

A transient lease-backend blip therefore permanently disables that shard type on that node for the lifetime of the ActorSystem.

Scope

This requires a lease to be configured for the coordinator singleton, i.e. pekko.cluster.sharding.coordinator-singleton.use-lease. That is a separate setting from pekko.cluster.sharding.use-lease, which per cluster-sharding/src/main/resources/reference.conf:423-426 applies only to per-shard leases:

  # Config path of the lease that each shard must acquire before starting entity actors
  # default is no lease
  # A lease can also be used for the singleton coordinator by settings it in the coordinator-singleton properties
  use-lease = ""

and reference.conf:369-370:

  # A lease can be configured in these settings for the coordinator singleton
  coordinator-singleton = ${pekko.cluster.singleton}

The chain

1. The termination message is ShardCoordinator.Internal.Terminate — cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/ClusterSharding.scala:833-837:

      context.actorOf(
        ClusterSingletonManager
          .props(singletonProps, terminationMessage = ShardCoordinator.Internal.Terminate, singletonSettings)

Note also :826, .withFinalStopMessage(_ == ShardCoordinator.Internal.Terminate) — the backoff supervisor treats this message as final, so the coordinator child is not restarted by backoff either.

2. Lease loss fires it while the node is still Up — cluster-tools/src/main/scala/org/apache/pekko/cluster/singleton/ClusterSingletonManager.scala:990-997:

    case Event(LeaseLost(reason), OldestData(singleton)) =>
      log.warning("Lease has been lost. Reason: {}. Terminating singleton and trying to re-acquire lease", reason)
      singleton match {
        case Some(s) =>
          s ! terminationMessage
          goto(AcquiringLease).using(AcquiringLeaseData(leaseRequestInProgress = false, singleton))
        case None =>
          tryAcquireLease()
      }

Nothing here consults the node's own member status. The intent, per the log message, is explicitly to re-acquire afterwards.

3. The coordinator shuts down the local region — cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/ShardCoordinator.scala:964-978:

  private def terminate(): Unit = {
    if (aliveRegions.exists(_.path.address.hasLocalScope) || gracefulShutdownInProgress.exists(
        _.path.address.hasLocalScope)) {
      aliveRegions
        .find(_.path.address.hasLocalScope)
        .foreach(region =>
          // region will get this from taking part in coordinated shutdown, but for good measure
          region ! ShardRegion.GracefulShutdown)

      log.debug("{}: Deferring coordinator termination until local region has terminated", typeName)
      waitingForLocalRegionToTerminate = true

terminate() has no notion of why it was called. The comment shows the assumption is a coordinated shutdown, which is not the case here.

4. The region drains and stops — ShardRegion.scala:935-957 sets gracefulShutdownInProgress = true (the CoordinatedShutdown.getShutdownReason() check at :945-952 only gates the timeout timer, not the shutdown itself), then :1109-1113:

  private def tryCompleteGracefulShutdownIfInProgress(): Unit =
    if (gracefulShutdownInProgress && shards.isEmpty && shardBuffers.isEmpty) {
      log.debug("{}: Completed graceful shutdown of region.", typeName)
      context.stop(self) // all shards have been rebalanced, complete graceful shutdown
    }

5. The coordinator then stops itself — ShardCoordinator.scala:1236-1243:

        aliveRegions -= ref
        allocateShardHomesForRememberEntities()
        if (ref.path.address.hasLocalScope && waitingForLocalRegionToTerminate) {
          // handoff optimization: singleton told coordinator to stop but we deferred stop until the local region
          // had completed the handoff
          log.debug("{}: Local region stopped, terminating coordinator", typeName)
          terminate()
        }

6. The dead region ref stays cached forever — ClusterSharding.scala:189 private val regions: ConcurrentHashMap[String, ActorRef]; start at :301-315:

      regions.get(typeName) match {
        case null => ... regions.put(typeName, shardRegion); shardRegion
        case ref => ref // already started, use cached ActorRef
      }

and shardRegion(typeName) at :644-654 returns the same cached ref. Nothing watches the region or removes the entry when it terminates.

Net effect: the coordinator singleton does come back — ClusterSingletonManager re-acquires the lease and starts a fresh one — but the local ShardRegion does not. ClusterSharding(system).shardRegion(typeName) keeps returning a terminated ref, so entity messages go to dead letters, with no log line connecting it to the earlier lease warning.

Existing coverage gap

cluster-sharding/src/test/scala/org/apache/pekko/cluster/sharding/ClusterShardingLeaseSpec.scala has a "recover if lease lost" case, and it does pass — but it exercises a shard lease, not the coordinator's:

    "recover if lease lost" in {
      region ! 4
      ...
      val testLease = leaseForShard(4)
      testLease.initialPromise.complete(Success(true))
      expectMsg(4)
      testLease.getCurrentCallback()(Option(LeaseFailed("oh dear")))
      awaitAssert({ region ! 4; expectMsg(4) }, max = 10.seconds)
    }

with leaseForShard / leaseNameFor at :109-114 resolving to s"${system.name}-shard-$typeName-$shardId". The spec's config sets pekko.cluster.sharding.use-lease, which does not give the coordinator singleton a lease. So the coordinator path is untested.

Suggested direction

ShardCoordinator.terminate() needs to distinguish "this node is leaving the cluster, so drain and stop the local region" from "the singleton is being restarted for a reason unrelated to membership". A guard along the lines of is this node actually leaving — cluster.isTerminated, or a CoordinatedShutdown reason is present, or the self member's status is Leaving/Exiting/Down — would let the lease-loss path stop only the coordinator and leave the local region running.

An independent, complementary hardening is to stop serving a terminated ref from the regions cache (ClusterSharding.scala:189) — watch the region and evict the entry so that a later start recreates it. That would limit the blast radius of any future path that stops a region unexpectedly. The same missing-watch pattern already affects the per-role replicatorByRole cache in the same guardian, which is tracked in #3691.

Reproduction recipe

Configure pekko.cluster.sharding.coordinator-singleton.use-lease = "test-lease" on top of the existing ClusterShardingLeaseSpec setup. The coordinator singleton's lease name is derived at ClusterSingletonManager.scala:521-525:

  val lease: Option[Lease] = settings.leaseSettings.map { settings =>
    val leaseName =
      if (settings.leaseName.isEmpty) s"${context.system.name}-singleton-${self.path}"
      else settings.leaseName

so it can be resolved from TestLeaseExt once the singleton has started. Then:

1. join a single-node cluster, start sharding, let the coordinator acquire its lease;
2. send a message and confirm an entity responds;
3. fire coordinatorLease.getCurrentCallback()(Option(LeaseFailed("blip")));
4. assert that ClusterSharding(system).shardRegion(typeName) still delivers — today it should instead be a terminated ref, with the message landing in dead letters.

Step 4 is the assertion that should fail before the fix.

Verification status

The chain above is traced statically; every link is cited from current main. I have not yet run the reproduction — the recipe is included so the fix can be developed against a failing test, per the contribution guideline that a bug-fix test must expose the issue first. If the reproduction does not fail as described, the analysis above is wrong somewhere and this should be closed.