Cluster: the inbound connection setup sends the registrations snapshot within heartbeat_interval

Open
  • Is something wrong?
#267 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
distributed-systems

Research direction

Read tellus/src/cluster/peer.rs around run_accepted (357–486), then compare the dialing-side snapshot flow around 1005–1010 and 1094. Check how the setup timeout encloses the registrations snapshot and where inbound readers are started. Done means registrations can be sent without the heartbeat timeout dropping the connection, and the timeout failure is logged at warning level with the peer.

Written by the indexing model from the issue text.

Description

bug

run_accepted (peer.rs 357-470) wraps the whole establishment of an inbound connection in timeout(heartbeat_interval, ..) (361): the handshake, the reply, the member snapshot, the reachability snapshot and, for a Member handshake, every frame of the registrations snapshot (450-457). Each frame goes out through write_all (quic.rs 495-516), which waits for QUIC flow control, and the registrations snapshot is bounded only by the caps of 2^20 registrations and 64 MiB (receptionist.rs 35-38). So the setup succeeds only if one heartbeat interval, 1 s by default, carries the node's whole registry.

What happens when it does not:

  • The timeout returns at 475 with debug!("timed out establishing inbound connection") and drops the connection. Nothing is logged above debug level: a frame which fails to send is a warning (454), a setup which runs out of time is not.
  • The dialer has already received the reply handshake, which is sent before the snapshots (429-436), so its own setup succeeds and its writers start. About a second later the connection is gone, which it notices at its next write, and it reconnects with backoff, for good, since a member's address is retried forever (772-783).
  • The accepting node never spawns read_streams for such a connection (486), so it records no heartbeat from the dialer beyond what it reads on its own dialed connection's setup frames (Liveness::track records one only when it first arms a detector, failure.rs 442-451). It marks every peer unreachable and, under KeepMajority, downs itself as the minority after after.
  • Connections established before the registry grew stay up, so the failure shows on later connections only: after a network blip, a restart, or when a new member joins.

How large is too large: a registration costs about 17 bytes plus its share of the key (group_len, receptionist.rs 770). One second carries about 100 MB at 1 Gbit/s, so on a LAN only a registry near the cap is affected; about 12 MB at 100 Mbit/s, some 600,000 registrations; about 1 MB at 10 Mbit/s, some 60,000. The dialing side has no such bound: it hands its snapshot to the lane's slot (lanes.rs 209), which the control writer drains outside any timeout (peer.rs 1005-1010, write_snapshot at 1094).

The timeout exists so that an unauthenticated dialer cannot pin the task (355-356), which the handshake exchange needs; the snapshots were swept into it. docs/cluster.md states the bound for the joiner reading the member snapshot (149-150) and calls the caps "safety caps, not a sizing target" (1225), without saying that connection setup must carry the snapshot in one heartbeat interval.

What I'd do:

  1. Send the registrations snapshot after the timeout scope, e.g. right before supersede_inbound_reader (486), or through a slot like the dialing side. The member and reachability snapshots are small and bounded and can stay inside.
  2. Log the setup timeout at warn level with the peer, like the other establishment failures around it.

That every setup timeout is heartbeat_interval is a separate point, in #268. I'm happy to do 1 and 2 if you agree, no hurry, after 0.6 and whenever it suits the backlog.

Dominant language
Rust
Stars
37
Forks
1
Avg merge
6h 36m
Merged PRs (30d)
123

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tellus-rs/tellus

All issues in tellus-rs/tellus

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.