Shard: a failed lease release in postStop is silently ignored
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 85/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- scala
- Domain
- distributed-systems, observability
Research direction
Start in cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/Shard.scala around lines 1236-1240 where postStop releases the lease. Read the surrounding Shard actor to see how logging is done, then apply the suggested onComplete callback on the release Future. Add a test in ShardWithLeaseSpec or ClusterShardingLeaseSpec using TestLease.setNextReleaseResult to assert a failed release logs an error. Done when the new warning/error logs fire and existing tests pass.
Written by the indexing model from the issue text.
Description
Summary
Shard releases its lease in postStop and discards the returned Future, so a failed release is never logged. This is the same class of problem as #3685 (fixed for ClusterSingletonManager in #3699), but in cluster sharding.
Evidence
cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/Shard.scala:1236-1240:
override def postStop(): Unit = {
passivateIntervalTask.foreach(_.cancel())
lease.foreach(_.release())
log.debug("{}: Shard [{}] shutting down", typeName, shardId)
}
Lease.release() returns Future[Boolean]. Here both a failed future and a false result (lease not released) are dropped silently.
Impact
Observability rather than correctness. When the lease backend is unavailable as a shard stops (hand-off, rebalance, region shutdown), the lease is not released and nothing says so. The shard's next allocation then waits until the lease times out, and the logs give no reason for the stall.
Suggested direction
The shard is stopping, so the result can't be piped back to self. Instead, log directly from the future's callback; the ActorLogging adapter is safe to use from another thread. For example:
lease.foreach { l =>
import context.dispatcher
val tn = typeName
val id = shardId
l.release().onComplete {
case Success(true) =>
case Success(false) => log.warning("{}: Lease for shard [{}] was not released", tn, id)
case Failure(t) => log.error(t, "{}: Failed to release lease for shard [{}]", tn, id)
}
}
Also consider guarding against a release() that throws synchronously, so the rest of postStop still runs.
Test
ShardWithLeaseSpec / ClusterShardingLeaseSpec use TestLease. #3699 adds TestLease.setNextReleaseResult, which allows a case asserting that a failed release logs an error when a shard stops.
- Dominant language
- Scala
- Stars
- 1.7k
- Forks
- 212
- Avg merge
- 21h 9m
- Merged PRs (30d)
- 98
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/pekko
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/pekko#1840 · 8 comments ·
Maintainers usually reply within 1 day
-
Coordinator singleton lease loss permanently stops the local ShardRegion while the node is still UpPossibly taken @He-Pin claimed this 1 day ago. Openbug cluster
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
enhancement flaky make-tests-easier-to-run
Difficulty 4/5 3-5 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
build enhancement github_actions
Difficulty 3/5 1-2 days Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
Similar issues
-
area:verification status:backlog
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
KD5UJC-XYZ/scarab#459 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
scalacenter/scaladex#1834 ·
-
PlatformSpecific.hasGreenThreads incorrectly returns false for JDK updates and non-GA versionsPossibly taken @hearnadam claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
zio/zio#11303 · 1 reaction · 2 assignees ·
Maintainers usually reply within 1 day
-
Disabled tools can still be called by the agentPossibly taken @shaynhornik claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 79/100
apache/texera#9122 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/pekko-persistence-jdbc#646 ·
Maintainers usually reply within 1 day