cmd/tailcat: a long-lived forward never re-registers, so new connections hang silently after the server restarts

Aperta
  • Qualcosa non va?
#159 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
go
Ambito
networking

Direzione di ricerca

Start by reading Client.up and Client.initLocked in tailcat.go, then trace onMeow and the dialing paths in cmd/tailcat/forward.go. Reproduce the server restart with a saved server key and a long-running forward client; the same-key ping workaround should restore it. Done when a running client recovers after a server restart, including re-advertising endpoints, and a regression test covers the recovery.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

A long-lived client process (tailcat forward, and likewise socks, the SSH ProxyCommand path, or an embedding program) stops working permanently once the server restarts, even when the server comes back on the same tailcat address. The client stays alive and keeps accepting on its local listeners, but every new connection through the tunnel hangs forever — no error, no log line, no recovery. Killing and restarting the client fixes it until the next server restart.

Root cause, as far as I can tell from the source: the meow bootstrap — the only thing that makes a server add a client to its netmap / WireGuard peer set — runs once per client process, and Client.up's latch is never invalidated:

// tailcat.go (main, 52fbad3) — Client.up
func (c *Client) up(ctx context.Context) error {
	if c.upDone.Load() {
		return nil
	}
	_, err := c.Ping(ctx)
	return err
}

The server's client set is in-memory only (locoBackend.clients, filled by onMeow; the state store is a mem.Store), so a restarted server knows nobody. A client that already latched upDone never announces itself again, and the new server drops its traffic.

Reproduction

Linux host, two temporary instances (server keeps a saved key so its address survives the restart):

tailcat genkey --key=/tmp/t/srv.private.json
tailcat genkey --client --key=/tmp/t/cli.private.json
python3 -m http.server 18099 --bind 127.0.0.1 &

tailcat --key=/tmp/t/srv.private.json serve 18099   # prints its tc… address; call it $TC
tailcat --key=/tmp/t/cli.private.json forward "$TC" 19099:18099

curl -m 5 http://127.0.0.1:19099/   # 200, forwarding works

# restart the server; same saved key, so the same address comes back
kill %serve; tailcat --key=/tmp/t/srv.private.json serve 18099

curl -m 90 http://127.0.0.1:19099/  # hangs, times out; forward logs nothing at all

Observed:

  • The long-running forward process is still alive, its listener still accepts TCP, and it prints only its startup # forwarding … line. Nothing is logged per connection, before or after the restart.
  • Waiting does not help: still broken 90 s after the restart, and still broken 20 s later.
  • tailcat ping "$TC" from a separate process using the same client node key immediately restores the running forward — the next curl returns 200. So the tunnel, DERP connection, and direct path are all fine; the missing piece is a fresh meow that re-adds the client on the server.
  • tailcat ping "$TC" from a process with a different client key does not help, as expected — the server keys clients by node public key.

Reproduced with the v0.7.0 release binary (Linux, both roles on one host, so both sides share a direct path and a DERP path). The code on main (52fbad3) is unchanged in the relevant paths.

Why the client can't notice

cmd/tailcat/forward.go:133 (forwardListener) dials with cl.DialTCPPort(ctx, …)/cl.DialTCP(ctx, …) where ctx is the process-lifetime context, so a dial has no deadline. A dial to a server that does not know us goes into the tunnel and is silently dropped; netstack keeps waiting, the connection hangs indefinitely, and ProxyConns never starts. Nothing in the client observes "the server forgot me".

Who is affected

Any client whose process outlives a server restart. Server restarts are routine: Restart=always under systemd, host reboots, serve config changes. In my case a Windows client's tailcat forward sat in a terminal, apparently healthy, and the forwarded port was dead until I Ctrl+C'd and re-ran it.

Suggested fixes

  1. Re-announce periodically. While a client is up, resend the meow every N seconds (or before a dial when the last meow is older than N seconds). A repeat meow is cheap: locoBackend.onMeow returns immediately when the key is already in b.clients (tailcat.go:1738). Note that the client only advertises its UDP endpoints once, in Client.initLocked's onMeowed (a sync.OnceFunc), so a re-meow path should also re-advertise endpoints, or the new server stays on DERP for traffic to the client.
  2. Bound the dial and retry once. Give the dial a timeout (5–10 s); on timeout, clear upDone, re-meow, and retry the dial once. That also converts the invisible hang into a log line, which is how this cost me an hour.
  3. Optionally, an explicit API to invalidate the registration (Client.ResetRegistration?) so embedders and the CLI can retry deliberately.

Happy to send a PR for (1) + (2) if you want a particular shape, or to test a branch.

Workaround for anyone hitting this

Give the client a durable identity and re-announce from a timer — that heals the running client without restarting it:

tailcat genkey --client --key=client-default   # forward and ping now share one node key
# then, every couple of minutes:
tailcat ping <tc-addr>

This only works if the long-lived client was started with that same key (so restart forward once after creating it). Without a durable key, the only cure is restarting the client process after each server restart. I verified both halves: same-key ping heals the stale forward instantly, different-key ping does not.

Thanks — tailcat is great otherwise; the meow bootstrap is a really nice design, it just needs to survive a server restart.

Lingua principale
Go
Stelle
8.8k
Fork
404
Merge medio
3h 48m
PR unite (30g)
1

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tailscale/tailcat

Tutte le issue di tailscale/tailcat

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.