Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

A frozen node is twice as expensive as an absent one, and marks its peer's completed work as failed

Offen
  • Stimmt etwas nicht?
#6 7 Kommentare 0 Reaktionen 1 zugewiesene Person Auf GitHub ansehen

Maintainer antworten meist innerhalb von 5 Tagen

@Noah-Tervalon-Nvidia arbeitet bereits daran.

Seit 11.9.2026.

Bewertung

Dieses Issue wurde noch nicht bewertet.

Beschreibung

Two observations from the same failure mode, measured on a two-node cluster (Ubuntu + RTX 3060, macOS, v0.1.1-463). Both matter for a cluster that includes a laptop, since a laptop that sleeps mid-request is exactly this shape: the socket is accepted, nothing answers.

Setup

24 concurrent chat/completions, max_tokens: 200, google/gemma-4-12b resident on both nodes.

Condition, same load Wall clock
Both nodes healthy 37.8 s
Single node, bypassing the router entirely 72.8 s
Peer frozen (SIGSTOP on its nvpair processes) 155.6 s

Every request returned 200 in all three. Reliability is not the problem — the cost is.

A frozen peer is 2.1× worse than not having it at all, and 4.1× worse than a healthy cluster. When I instead killed the peer's processes outright, the same load finished in 28.3 s with no failures: a closed socket gives an immediate RST and the proxy moves on in milliseconds. It is specifically the accepted-but-silent socket that hurts.

For context on the eviction path: after the peer was killed, discovery had dropped it from the node list within 75 s — I sampled too coarsely to pin it more precisely. During that window each request pays its own connection timeout.

The second half: false failures

When the frozen peer resumed, its job history marked several of the other node's workloads as

Failed — node serveri stopped reporting this workload

Those workloads had all completed successfully; the requesting node returned 200 for all 24. The peer's view had simply gone stale while it was stopped, and on resume it reaped them as failed. The timestamps match the resume to the second.

That leaves the job history — the only place a user sees what the cluster did — showing failures that did not happen, precisely in the situation where something unusual was going on and they would be looking at it.

Not filed as a defect

Neither is a crash and both may be deliberate. But if a laptop is a first-class node, asleep with the socket accepted and nothing answering is its most likely state, not an edge case. Two directions that would help, if either is welcome:

  • a connection-level timeout distinctly shorter than the request timeout, so an unresponsive candidate is abandoned quickly while a slow-but-live generation is not
  • reaping a peer's workloads as unknown rather than failed when the gap was on our side

Happy to open a PR for either if you have a preferred shape.

Vorherrschende Sprache
Go
Sterne
1.6k
Forks
266
Ø Merge
3 T. 6 Std.
Gemergte PRs (30 T.)
22

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus NVIDIA/Personal-AI-Router

Alle Issues in NVIDIA/Personal-AI-Router

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.