A frozen node is twice as expensive as an absent one, and marks its peer's completed work as failed
维护者通常 2 天内回复
@Noah-Tervalon-Nvidia 已经在做这个了。
开始于 2026年9月11日。
评估
这个 Issue 还没有评估数据。
描述
Two observations from the same failure mode, measured on a two-node cluster (Ubuntu + RTX 3060, macOS, v0.1.1-463). Both matter for a cluster that includes a laptop, since a laptop that sleeps mid-request is exactly this shape: the socket is accepted, nothing answers.
Setup
24 concurrent chat/completions, max_tokens: 200, google/gemma-4-12b resident on both nodes.
| Condition, same load | Wall clock |
|---|---|
| Both nodes healthy | 37.8 s |
| Single node, bypassing the router entirely | 72.8 s |
Peer frozen (SIGSTOP on its nvpair processes) |
155.6 s |
Every request returned 200 in all three. Reliability is not the problem — the cost is.
A frozen peer is 2.1× worse than not having it at all, and 4.1× worse than a healthy cluster. When I instead killed the peer's processes outright, the same load finished in 28.3 s with no failures: a closed socket gives an immediate RST and the proxy moves on in milliseconds. It is specifically the accepted-but-silent socket that hurts.
For context on the eviction path: after the peer was killed, discovery had dropped it from the node list within 75 s — I sampled too coarsely to pin it more precisely. During that window each request pays its own connection timeout.
The second half: false failures
When the frozen peer resumed, its job history marked several of the other node's workloads as
Failed — node serveri stopped reporting this workload
Those workloads had all completed successfully; the requesting node returned 200 for all 24. The peer's view had simply gone stale while it was stopped, and on resume it reaped them as failed. The timestamps match the resume to the second.
That leaves the job history — the only place a user sees what the cluster did — showing failures that did not happen, precisely in the situation where something unusual was going on and they would be looking at it.
Not filed as a defect
Neither is a crash and both may be deliberate. But if a laptop is a first-class node, asleep with the socket accepted and nothing answering is its most likely state, not an edge case. Two directions that would help, if either is welcome:
- a connection-level timeout distinctly shorter than the request timeout, so an unresponsive candidate is abandoned quickly while a slow-but-live generation is not
- reaping a peer's workloads as
unknownrather thanfailedwhen the gap was on our side
Happy to open a PR for either if you have a preferred shape.
- 主要语言
- Go
- 星标
- 1.6k
- 派生
- 277
- 平均合并
- 4 天 18 小时
- 30 天内合并 PR
- 33
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
NVIDIA/Personal-AI-Router 的其他 Issue
-
难度 5/5 一周以上 新手友好度 20/100
NVIDIA/Personal-AI-Router#165 ·
维护者通常 2 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 46/100
NVIDIA/Personal-AI-Router#164 ·
维护者通常 2 天内回复
-
enhancement
难度 4/5 一周以上 新手友好度 12/100
NVIDIA/Personal-AI-Router#162 ·
维护者通常 2 天内回复
-
enhancement
难度 4/5 3-5 天 新手友好度 25/100
NVIDIA/Personal-AI-Router#154 · 1 条评论 ·
维护者通常 2 天内回复
-
[Feature]: [Ollama] Alert the user about model pull failures可能已有人在做 @ckelseynv 于 5 天前认领。 未关闭enhancement
难度 3/5 1-2 天 新手友好度 56/100
NVIDIA/Personal-AI-Router#152 · 1 条评论 · 已指派 1 人 ·
维护者通常 2 天内回复
查看 NVIDIA/Personal-AI-Router 的全部 Issue
相似的 Issue
-
copilot-api domain:platform go
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
docs: clarify gh api -f vs -F and @-file expansion in the gh skill可能已有人在做 @MatiasOlivera 今天认领。 未关闭enhancement
难度 2/5 半天 新手友好度 62/100
维护者通常 1 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 88/100
aws/aws-advanced-go-wrapper#612 ·
-
Say in the changelog that the text-index update cannot be rolled back可能已有人在做 @praetorianer777 今天认领。 未关闭
难度 1/5 1 小时以内 新手友好度 88/100
praetorianer777/GoTome#275 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
SAP/crossplane-provider-btp#1118 ·
维护者通常 1 天内回复