Cluster peer stuck offline when paired over non-multicast link (Tailscale / VPN) — mDNS does not traverse tailscale0
维护者通常 5 天内回复
@ckelseynv 已经在做这个了。
开始于 2026年9月18日。
评估
这个 Issue 还没有评估数据。
描述
Cluster member stuck "offline" when paired over a non-multicast link (Tailscale / VPN)
Summary
After a successful PIN pairing between two nodes connected only via a non-multicast
link (Tailscale, ZeroTier, Hamachi, corporate VPN), the peer is permanently shown
as "offline" in the Overview tab and members.json lastSeen stays null,
even though every cluster TCP port (14318, 14319, 14321) is reachable both ways
and the broker log shows nvpair-ui-broker and nvpair-cluster-manager running.
Reproduction
- Two machines on the same Tailscale network (e.g.
100.100.108.73and
100.121.35.74), neither on the same LAN. - Settings → Cluster → Add node → invite by IP (Tailscale IP) → accept PIN.
Pairing completes; both members showstate: "member". - Engine-manager logs repeat forever:
remote engine status for <peer-uuid> unavailable: -32000: node <peer-uuid> is not a discovered ec peer. members.jsonfor both members showslastSeen: null, even after minutes.- The UI Overview tab never reports the peer as online.
Root cause
Presence / "online" status is driven by the mDNS discovery layer
(nvpair-node-scanner listening for _nvpair-node._tcp announcements), not by
the cluster-manager's roster probe. Multicast DNS (224.0.0.251) does not
traverse Tailscale's userspace WireGuard tunnels — both nodes send mDNS queries
on tailscale0 (mdns send: ... iface=tailscale0 ip=100.100.108.73) but no
peers' announcements ever arrive, so the peer never enters the discovery store.
Once absent from discovery, nvpair-engine-manager rejects every dial with
-32000: not a discovered ec peer, and nvpair-cluster-manager's lastSeen
timestamp is never updated.
Workaround that works today
Seed configs/manual-nodes.json (Electron-side persistence in
desktop/src/electron/service-bridge/manual-nodes-store.ts) and restart the app.
The supervisor's replayManualNodes() re-adds the entry via node/add over the
broker; nvpair-manual-nodes probes the peer directly over Tailscale every
~10 s, emits node/discovered, the broker merges it into the discovery
snapshot, and engine-manager stops rejecting the dial.
[
{
"id": "dabeast",
"address": "100.121.35.74",
"name": "dabeast"
}
]
After app restart, logs confirm:
[nvpair-manual-nodes] INFO manual node state changed node_id=dabeast addr=100.121.35.74 ollama_up=true node_info_up=true models=13 gpus=2discovery:nodes-changedcarries the peer with full model inventory- Overview tab flips the peer to online
Suggested fixes (any one would address the root cause)
- Document the workaround. Surface manual-node entry in the desktop UI
(currentlySettings → Cluster → Add nodedoes PIN-pairing only — see
desktop/src/ui/components/AddNodeModal.tsx— there is no manual-node
registration UI). - Treat manual-nodes entries as authoritative for presence. When a peer
was paired by IP and not seen via mDNS, fall back tonvpair-manual-nodes
status for heartbeat /lastSeenupdates. - Add a Tailscale-aware discovery path. Either (a) a Tailscale MagicDNS /
unicast DNS-SD fallback when mDNS browse times out, or (b) a peer relay
over the cluster's existing mTLS channel so each node advertises its
cluster peers to the others. - Avoid the crash-on-add churn. Adding a manual node today restarts
nvpair-cluster-manager; combined with the worker's in-memory state loss,
this makes the pairing flow itself brittle.
Logs
14:23:05 [nvpair-manual-nodes] INFO log level changed level=debug
14:39:57 [nvpair-manual-nodes] INFO using stdio transport
14:40:30 [nvpair-ui-broker] ERROR nvpair-errors crashed; service-error pipeline
unavailable until it restarts attempt=3
14:40:32 [nvpair-ui-broker] WARN supervisor: worker exited unexpectedly
worker=cluster-manager attempt=3
14:40:42 [nvpair-engine-manager] WARN remote engine status for
18e94d12-e71f-42c9-8c07-355f68bb2fb8 unavailable: -32000: node
18e94d12-e71f-42c9-8c07-355f68bb2fb8 is not a discovered ec peer
(Repeats every ~20 s. Discovery nodes-changed lists only linuxbeast.)
Environment
- nvpair 0.1.1 (pacman build from
release/0.1.1/linux/NVPAIR-Setup-0.1.1-x64.pacman) - Linux 7.2.5-3-omarchy, kernel 7.2.5-3-omarchy
- Tailscale 100.x on both nodes, direct peer-to-peer connection (
active; direct) - 13 models on peer, 2 GPUs
- 主要语言
- Go
- 星标
- 1.6k
- 派生
- 266
- 平均合并
- 3 天 6 小时
- 30 天内合并 PR
- 22
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
NVIDIA/Personal-AI-Router 的其他 Issue
-
难度 5/5 一周以上 新手友好度 20/100
NVIDIA/Personal-AI-Router#165 ·
维护者通常 5 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 46/100
NVIDIA/Personal-AI-Router#164 ·
维护者通常 5 天内回复
-
enhancement
难度 4/5 一周以上 新手友好度 12/100
NVIDIA/Personal-AI-Router#162 ·
维护者通常 5 天内回复
-
enhancement
难度 4/5 3-5 天 新手友好度 25/100
NVIDIA/Personal-AI-Router#154 · 1 条评论 ·
维护者通常 5 天内回复
-
[Feature]: [Ollama] Alert the user about model pull failures可能已有人在做 @ckelseynv 于 4 天前认领。 未关闭enhancement
难度 3/5 1-2 天 新手友好度 56/100
NVIDIA/Personal-AI-Router#152 · 1 条评论 · 已指派 1 人 ·
维护者通常 5 天内回复
查看 NVIDIA/Personal-AI-Router 的全部 Issue
相似的 Issue
-
Action Required: Fix Renovate Configuration可能已有人在做 @thompson-tomo 今天认领。 未关闭
难度 1/5 1 小时以内 新手友好度 78/100
open-telemetry/opentelemetry-collector#16170 ·
维护者通常 1 天内回复
-
automation awf compiler specifications
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
state:triage-needed
难度 2/5 1-3 小时 新手友好度 65/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 80/100
puppe1990/amarra-cais#342 ·
维护者通常 1 天内回复
-
replay: a run cancelled during environment setup is logged as an ERROR (Instrument drops the cancellation)可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复