Local node's GPU/CPU/memory enrichment is never refreshed after startup
維護者通常 5 天內回覆
評估
這個 Issue 還沒有評估資料。
描述
The discovery directory's enrichment for the local node is captured once and then frozen. Peers converge; self does not.
What I measured
Ubuntu node with an RTX 3060, paired with a macOS node, v0.1.1-463 from the .deb.
I loaded a 7 GB model into LM Studio with full GPU offload. nvidia-smi went from 121 MiB to 9329 MiB. node-info served the new figure within its usual cadence — curl 127.0.0.1:14318/v1/node-info returned vram_used_bytes: 9782165504 with telemetryValid: true, msSince: 280. The desktop UI's VRAM graph showed the step too, because it reads the node directly.
The discovery snapshot did not move. discovery:get-nodes kept returning vram_used_bytes: 126877696 — the value from daemon start — for as long as I watched, checked at 20 s intervals over several minutes. Restarting the broker made it correct again, until the next change.
Why
onBrowse deliberately skips our own entry (daemon.go: "Never let a browse event touch our own entry. Self is registry-driven"), so the path that re-enriches every peer on every browse never runs for self. publishSelf — the only thing that enriches self — is called at startup, on service (un)register, and after an identity/address change. refreshPeersLoop refreshes advertised addresses, the mesh, cluster identity and models on its 15 s tick, but nothing re-reads self's node-info.
Worth noting that the comment above noteNodeInfo already states the intended behaviour:
The enrichment sweep repeats every peerRefreshInterval for as long as a node is advertised
That holds for peers, not for self.
Who sees it
Only consumers of the discovery snapshot — which includes nvpair-job-scheduler, since the broker fans the same snapshot to it. The UI is unaffected, which is why this is easy to miss.
Repro
- Pair two nodes; note the local node's
gpus[].vram_used_bytesviadiscovery:get-nodes. - Load a large model with GPU offload on that node.
curl 127.0.0.1:14318/v1/node-info— fresh.discovery:get-nodes— unchanged, indefinitely.
A fix that works
Calling d.publishSelf() on the refreshPeersLoop tick converges the directory within 20 s. Verified live in both directions, loading and unloading. It does mean one self node-updated per tick, since publishSelf emits unconditionally; the tidier alternative is an applyInfo path with a changed check, mirroring applyModels. Happy to open a PR for either shape if useful.
- 主要語言
- Go
- 星號
- 1.6k
- 分支
- 266
- 平均合併
- 3 天 6 小時
- 30 天內合併 PR
- 22
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
NVIDIA/Personal-AI-Router 的其他 Issue
-
難度 5/5 一週以上 新手友好度 20/100
NVIDIA/Personal-AI-Router#165 ·
維護者通常 5 天內回覆
-
bug
難度 4/5 3-5 天 新手友好度 46/100
NVIDIA/Personal-AI-Router#164 ·
維護者通常 5 天內回覆
-
enhancement
難度 4/5 一週以上 新手友好度 12/100
NVIDIA/Personal-AI-Router#162 ·
維護者通常 5 天內回覆
-
enhancement
難度 4/5 3-5 天 新手友好度 25/100
NVIDIA/Personal-AI-Router#154 · 1 則留言 ·
維護者通常 5 天內回覆
-
[Feature]: [Ollama] Alert the user about model pull failures可能已有人在做 @ckelseynv 於 4 天前認領。 未關閉enhancement
難度 3/5 1-2 天 新手友好度 56/100
NVIDIA/Personal-AI-Router#152 · 1 則留言 · 已指派 1 人 ·
維護者通常 5 天內回覆
查看 NVIDIA/Personal-AI-Router 的全部 Issue
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 85/100
維護者通常 1 天內回覆
-
kind/bug
難度 1/5 1-3 小時 新手友好度 85/100
kubernetes-sigs/vsphere-csi-driver#4351 ·
維護者通常 1 天內回覆
-
component/gpu-kubelet-plugin component/helm needs-triage
難度 2/5 1-3 小時 新手友好度 62/100
kubernetes-sigs/dra-driver-nvidia-gpu#1559 ·
維護者通常 1 天內回覆
-
workflow: redundant goroutine in scheduleRetry可能已有人在做 @moodbased 今天認領。 未關閉
難度 2/5 1-3 小時 新手友好度 70/100
維護者通常 2 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
JuliusBrussee/caveman#1266 · 2 則留言 ·
維護者通常 1 天內回覆