A shard can fail to send replication heads after an expiry event and a brief not-ready state

The sequence centres on Group.connect/2, expire_replica_peer, send_replica_heads, and the peer_connect, peer_connect_ack and replica_cluster_open handlers, with route_ready? and epoch deduplication deciding whether heads get resent. Start by reading the TLA+ model the reporter cites next to the readiness check in the code, since their mismatch is the root cause. Done means a repeated peer_connect at the same epoch after a not-ready ack still causes heads to be sent, which needs maintainer agreement on the intended protocol first.

Hello, this is the second issue I identified while experimenting with DST on the Group repo. This one is more involved than #32 , and I was only able to get to about 80% understanding myself. I hope the maintainers find it useful rather than noise.

Description

The summary of the issue is that a shard can fail to send its replication heads after an expiry event and a brief not-ready state. One trigger is a well-timed call to Group.connect/2.

We have a cluster with a, b, and c nodes and multiple shards configured.

Consider this sequence of events:

1. A c shard-0 (c0) is sending heartbeat messages to the corresponding shard b0, but each one remains undelivered. For example, these can be dropped when the dist port is busy as a result of the :nosuspend on the send.
2. b0 invokes expire_replica_peer after a series of missed heartbeats. b0's belief of c0 data is wiped locally, including the replication cursors for c’s streams and the entry in remote_shards. b0 begins rediscovery, which includes new peer_connect messages. It bumps its probe epoch for c0 to N.
3. c0 still holds the same info it had previous. Namely, c0 has no pending heads for b, since each was acked until this point.
4. On b, we execute Group.connect(:my_app, "blue"). The revision is bumped to R+1.
5. b0's anti-entropy timer puts the message on the b0 mailbox.
6. b puts :cluster_connect on the b0 mailbox.
7. b0 handles the anti-entropy message, sends a peer_connect to c0 with probe epoch N. The revision, read from ets, is R+1.
8. b0 handles the :cluster_connect, sends replica_cluster_open to c0, revision R+1.
9. c0 handles the peer_connect probe. It has only seen revision R, so route_ready? is false. c0 records epoch N as seen, and acks with peer_connect_ack(N, false). Because the probe is not ready, c0 does not call send_replica_heads.
10. c0 handles the replica_cluster_open. At this point, a future peer_connect request on a new epoch would be responded to with "ready", and would also resend heads. However, we'll see that a peer_connect with the same epoch N will not.
11. b0 receives the "not-ready" ack and at some future time the process sends another peer_connect, still with epoch N.
12. c0 receives the peer_connect at epoch N, decides epoch N has already been seen (:duplicate) and acks with peer_connect_ack(N, true). No heads are sent.
13. b0 receives and stops probing. Nothing will ever send c0’s heads to b0 again.

In the TLA+ model, there is a mismatch between what the model describes and what the code implements. The code has an extra readiness check that exposes the bug.

Testing performed on main at 08b5a5.

DST details

Config:

• Injected faults: heartbeats dropped with probability 0.7
• 6 of 600 seeds failed in the sweep. 1 of those 6 is the seqeunce described above. The others appear to have a similar pattern

Code:

• In order to instrument Group for DST with eta, I've had to make changes locally. The changes are focused on allowing Group to run multiple nodes within the same VM. The code is in my fork here.
• Details about the issue were confirmed on unmodified main outside of DST, but not a full runtime reproduction.