cmd/tailcat: a long-lived forward never re-registers, so new connections hang silently after the server restarts

Đang mở
  • Có gì không ổn?
#159 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
go
Lĩnh vực
networking

Hướng nghiên cứu

Start by reading Client.up and Client.initLocked in tailcat.go, then trace onMeow and the dialing paths in cmd/tailcat/forward.go. Reproduce the server restart with a saved server key and a long-running forward client; the same-key ping workaround should restore it. Done when a running client recovers after a server restart, including re-advertising endpoints, and a regression test covers the recovery.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Summary

A long-lived client process (tailcat forward, and likewise socks, the SSH ProxyCommand path, or an embedding program) stops working permanently once the server restarts, even when the server comes back on the same tailcat address. The client stays alive and keeps accepting on its local listeners, but every new connection through the tunnel hangs forever — no error, no log line, no recovery. Killing and restarting the client fixes it until the next server restart.

Root cause, as far as I can tell from the source: the meow bootstrap — the only thing that makes a server add a client to its netmap / WireGuard peer set — runs once per client process, and Client.up's latch is never invalidated:

// tailcat.go (main, 52fbad3) — Client.up
func (c *Client) up(ctx context.Context) error {
	if c.upDone.Load() {
		return nil
	}
	_, err := c.Ping(ctx)
	return err
}

The server's client set is in-memory only (locoBackend.clients, filled by onMeow; the state store is a mem.Store), so a restarted server knows nobody. A client that already latched upDone never announces itself again, and the new server drops its traffic.

Reproduction

Linux host, two temporary instances (server keeps a saved key so its address survives the restart):

tailcat genkey --key=/tmp/t/srv.private.json
tailcat genkey --client --key=/tmp/t/cli.private.json
python3 -m http.server 18099 --bind 127.0.0.1 &

tailcat --key=/tmp/t/srv.private.json serve 18099   # prints its tc… address; call it $TC
tailcat --key=/tmp/t/cli.private.json forward "$TC" 19099:18099

curl -m 5 http://127.0.0.1:19099/   # 200, forwarding works

# restart the server; same saved key, so the same address comes back
kill %serve; tailcat --key=/tmp/t/srv.private.json serve 18099

curl -m 90 http://127.0.0.1:19099/  # hangs, times out; forward logs nothing at all

Observed:

  • The long-running forward process is still alive, its listener still accepts TCP, and it prints only its startup # forwarding … line. Nothing is logged per connection, before or after the restart.
  • Waiting does not help: still broken 90 s after the restart, and still broken 20 s later.
  • tailcat ping "$TC" from a separate process using the same client node key immediately restores the running forward — the next curl returns 200. So the tunnel, DERP connection, and direct path are all fine; the missing piece is a fresh meow that re-adds the client on the server.
  • tailcat ping "$TC" from a process with a different client key does not help, as expected — the server keys clients by node public key.

Reproduced with the v0.7.0 release binary (Linux, both roles on one host, so both sides share a direct path and a DERP path). The code on main (52fbad3) is unchanged in the relevant paths.

Why the client can't notice

cmd/tailcat/forward.go:133 (forwardListener) dials with cl.DialTCPPort(ctx, …)/cl.DialTCP(ctx, …) where ctx is the process-lifetime context, so a dial has no deadline. A dial to a server that does not know us goes into the tunnel and is silently dropped; netstack keeps waiting, the connection hangs indefinitely, and ProxyConns never starts. Nothing in the client observes "the server forgot me".

Who is affected

Any client whose process outlives a server restart. Server restarts are routine: Restart=always under systemd, host reboots, serve config changes. In my case a Windows client's tailcat forward sat in a terminal, apparently healthy, and the forwarded port was dead until I Ctrl+C'd and re-ran it.

Suggested fixes

  1. Re-announce periodically. While a client is up, resend the meow every N seconds (or before a dial when the last meow is older than N seconds). A repeat meow is cheap: locoBackend.onMeow returns immediately when the key is already in b.clients (tailcat.go:1738). Note that the client only advertises its UDP endpoints once, in Client.initLocked's onMeowed (a sync.OnceFunc), so a re-meow path should also re-advertise endpoints, or the new server stays on DERP for traffic to the client.
  2. Bound the dial and retry once. Give the dial a timeout (5–10 s); on timeout, clear upDone, re-meow, and retry the dial once. That also converts the invisible hang into a log line, which is how this cost me an hour.
  3. Optionally, an explicit API to invalidate the registration (Client.ResetRegistration?) so embedders and the CLI can retry deliberately.

Happy to send a PR for (1) + (2) if you want a particular shape, or to test a branch.

Workaround for anyone hitting this

Give the client a durable identity and re-announce from a timer — that heals the running client without restarting it:

tailcat genkey --client --key=client-default   # forward and ping now share one node key
# then, every couple of minutes:
tailcat ping <tc-addr>

This only works if the long-lived client was started with that same key (so restart forward once after creating it). Without a durable key, the only cure is restarting the client process after each server restart. I verified both halves: same-key ping heals the stale forward instantly, different-key ping does not.

Thanks — tailcat is great otherwise; the meow bootstrap is a really nice design, it just needs to survive a server restart.

Ngôn ngữ chính
Go
Star
8.8k
Fork
404
Merge trung bình
3 giờ 48 phút
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của tailscale/tailcat

Tất cả issue của tailscale/tailcat

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.