Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

A frozen node is twice as expensive as an absent one, and marks its peer's completed work as failed

Aberta
  • Algo errado?
#6 7 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 5 dias

@Noah-Tervalon-Nvidia já está trabalhando nisso.

Desde 11/9/2026.

Avaliação

Esta issue ainda não foi avaliada.

Descrição

Two observations from the same failure mode, measured on a two-node cluster (Ubuntu + RTX 3060, macOS, v0.1.1-463). Both matter for a cluster that includes a laptop, since a laptop that sleeps mid-request is exactly this shape: the socket is accepted, nothing answers.

Setup

24 concurrent chat/completions, max_tokens: 200, google/gemma-4-12b resident on both nodes.

Condition, same load Wall clock
Both nodes healthy 37.8 s
Single node, bypassing the router entirely 72.8 s
Peer frozen (SIGSTOP on its nvpair processes) 155.6 s

Every request returned 200 in all three. Reliability is not the problem — the cost is.

A frozen peer is 2.1× worse than not having it at all, and 4.1× worse than a healthy cluster. When I instead killed the peer's processes outright, the same load finished in 28.3 s with no failures: a closed socket gives an immediate RST and the proxy moves on in milliseconds. It is specifically the accepted-but-silent socket that hurts.

For context on the eviction path: after the peer was killed, discovery had dropped it from the node list within 75 s — I sampled too coarsely to pin it more precisely. During that window each request pays its own connection timeout.

The second half: false failures

When the frozen peer resumed, its job history marked several of the other node's workloads as

Failed — node serveri stopped reporting this workload

Those workloads had all completed successfully; the requesting node returned 200 for all 24. The peer's view had simply gone stale while it was stopped, and on resume it reaped them as failed. The timestamps match the resume to the second.

That leaves the job history — the only place a user sees what the cluster did — showing failures that did not happen, precisely in the situation where something unusual was going on and they would be looking at it.

Not filed as a defect

Neither is a crash and both may be deliberate. But if a laptop is a first-class node, asleep with the socket accepted and nothing answering is its most likely state, not an edge case. Two directions that would help, if either is welcome:

  • a connection-level timeout distinctly shorter than the request timeout, so an unresponsive candidate is abandoned quickly while a slow-but-live generation is not
  • reaping a peer's workloads as unknown rather than failed when the gap was on our side

Happy to open a PR for either if you have a preferred shape.

Linguagem predominante
Go
Estrelas
1.6k
Forks
266
Merge médio
3d 6h
PRs com merge (30d)
22

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de NVIDIA/Personal-AI-Router

Todas as issues de NVIDIA/Personal-AI-Router

Issues semelhantes

Mais issues de Go

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.