Peer relay: the first relay path that works is kept for up to a minute, even when a much closer relay is a candidate

Open
  • Is something wrong?
#22,321 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go
Domain
networking

Research direction

Start with allocateAllServersRunLoop and candidatePeerRelay.derpHomeRegionID, which the issue identifies as the allocation and candidate-ranking entry points. Trace how the endpoint adopts the first completed handshake and how upgradeUDPRelayInterval revisits relay choices. The issue gives three possible approaches; done means nearby relay candidates are considered promptly after restart so the pair avoids staying on a much higher-latency relay for about a minute.

Written by the indexing model from the issue text.

Description

bug connectivity peer-relays
What is the issue?

When a node may use several peer relays in different regions, relayManager allocates on all of them at once (allocateAllServersRunLoop) and the endpoint adopts whichever relay path completes a handshake first. A lower-latency relay is only looked for on the next upgradeUDPRelayInterval tick, one minute later.

After either end of a pair restarts, a pair that is ~1 ms from a relay in its own region sometimes lands on a relay on another continent (~200 ms RTT) and stays there for up to ~60 s. For that window the relayed path is much slower than DERP would have been.

Steps to reproduce
  1. Run two peer relays far apart (we had us-central1 and Hong Kong), with tailscale.com/cap/relay granted on both to every node.
  2. Put two nodes in us-central1 behind endpoint-dependent NAT, so they can't form a direct path.
  3. Restart one of them. Its log sometimes shows now using <far-relay-ip>:<port>:vni, and tailscale debug peer-relay-sessions on the far relay lists the pair. It moves to the near relay about a minute later.
What did you expect?

Relay choice should account for latency the first time, not only on the 1-minute upgrade. Some options:

  • wait briefly for more allocations to answer before adopting one;
  • rank candidates by DERP home region, which candidatePeerRelay.derpHomeRegionID already tracks;
  • re-check sooner when the chosen relay path's latency is far worse than the other candidates' allocations.
Workaround

Scope the cap/relay grants by region, so a node only allocates on relays near it.

Version

v1.104.x clients and relays. The relays are k8s-operator PeerRelay resources and plain tailscaled --relay-server-port VMs.

Dominant language
Go
Stars
38.8k
Forks
3.4k
Avg merge
2d 7h
Merged PRs (30d)
194

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tailscale/tailscale

All issues in tailscale/tailscale

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.