Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

[Feature]: External / heterogeneous scheduling hooks and richer placement signals

Aberta
  • Algo errado?
#7 3 comentários 0 reações 1 responsável Ver no GitHub

Mantenedores costumam responder em até 5 dias

@sherief-nv já está trabalhando nisso.

Desde 9/10/2026.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
25/100
Tipo de issue
Funcionalidade
Clareza
Precisa de esclarecimento
Status de atividade
Ativa
Stack de tecnologia
go
Domínio
api, backend, distributed-systems, observability

Direção de pesquisa

A issue não identifica arquivos de implementação nem entry points; comece localizando o código de routing e scheduling e os testes de integração existentes. Divida a proposta em capacidades com escopo definido antes de alterar qualquer coisa. Para considerar o trabalho concluído, é necessário preservar o comportamento de fallback nativo, isolar as dicas de requests concorrentes, impor as verificações de trust e eligibility e validar o placement e a telemetria por meio dos cenários listados de clusters heterogêneos.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement
Area

Routing and scheduling

User problem

PAIR provides a strong transport, discovery, and inference data plane for distributed local inference, but heterogeneous clusters need richer placement signals than model ownership, queue depth, and coarse GPU utilization alone.

Today, an external scheduler cannot safely influence placement for one request without changing a persistent or proxy-wide pin, which can introduce races under concurrent traffic. Nodes that own the same model can also have very different practical runtime envelopes: characterized context limits, memory requirements, full-GPU versus CPU/offload behavior, model-load time, prompt/generation throughput, and reliability.

Requests themselves vary substantially in cost based on prompt length, requested context, expected output, model size, and modality. PAIR also has useful model-state visibility, but warm-model affinity and host-wide pressure from other compute workloads are not fully represented in routing decisions. Finally, operators need explainable placement metadata without exposing prompt content.

Desired outcome

PAIR should optionally support the following while preserving its native distributed scheduler as the self-contained default and fallback:

  1. Request-scoped placement hints such as a preferred node or ranked candidate list. PAIR would continue to enforce model ownership, availability, trust, and normal safety checks.
  2. Per-model/per-node runtime profiles learned empirically, including characterized context sizes, VRAM/RAM requirements, GPU/offload behavior, load time, prompt TPS, generation TPS, and stability history.
  3. Weighted request-cost estimation using inputs such as prompt tokens, requested context, expected output length, model size, modality, and historical model/node cost.
  4. Warm-model affinity when a suitable model is already resident and resource pressure does not favor another node.
  5. An optional external placement-authority hook, with automatic fallback to PAIR's native scheduler if that authority is unavailable.
  6. Host-wide admission signals so external GPU/CPU/memory pressure can influence placement without PAIR needing to manage those workloads.
  7. Explainable routing telemetry showing candidates, exclusions, selected node, selection reason, and failovers, without logging prompt content.
  8. Optional telemetry hardening under the same cluster trust/mTLS model used by other PAIR services.

The observable result would be safer and more predictable scheduling across heterogeneous nodes, lower model load/unload churn and tail latency, and easier integration with higher-level orchestration systems while PAIR remains the inference transport and failover layer.

Guidottavio D’Assisti
[email protected]

Alternatives considered
  • Continue using PAIR's native scheduler based primarily on queued/running workload and GPU pressure. This remains a valuable default and fallback, but it cannot distinguish requests or model/node combinations with radically different practical costs.
  • Use the existing manual node pin. A persistent or proxy-wide pin is not sufficient for concurrent, per-request placement and may create global-state races.
  • Route directly to inference engines from an external orchestrator. This would duplicate PAIR's model eligibility, discovery, mTLS transport, streaming, and failover responsibilities.
  • Infer capacity only from hardware specifications. Empirically learned runtime envelopes are more representative of real behavior across models, contexts, backends, and offload modes.
Compatibility and security implications

All proposed capabilities should be optional and backward-compatible. Requests without placement hints should retain current routing behavior, and an unavailable external authority should fall back to PAIR's native scheduler.

Placement hints must remain advisory to PAIR's eligibility, availability, and trust checks; they must not bypass model ownership, pairing, mTLS, or failover safety. Unknown or unauthorized node identifiers should be rejected or ignored according to a documented policy.

Routing telemetry should exclude prompt and response content and avoid exposing unnecessary host details. If telemetry protection is extended, existing trusted-local deployments should have a documented migration path. API extensions should use optional headers or namespaced fields so existing OpenAI- and Ollama-compatible clients continue to work unchanged.

Validation approach

Maintainers could validate the outcome with automated integration tests covering:

  • a request-scoped preferred node being selected only when it is reachable, trusted, and owns the requested model;
  • deterministic fallback when the preferred node or external authority is unavailable;
  • concurrent requests carrying different placement hints without cross-request interference;
  • ranking changes driven by distinct model/node runtime profiles, weighted request costs, and warm-model state;
  • host-wide admission signals excluding a pressured node and recovering when pressure clears;
  • OpenAI- and Ollama-compatible requests without new hints preserving existing behavior;
  • routing telemetry explaining candidates, exclusions, selection, and failovers while containing no prompt/response content;
  • unauthorized or malformed hints failing safely and telemetry endpoints enforcing the configured trust model.

A small heterogeneous test matrix with the same model available on differently characterized nodes could verify improved placement and reduced cold-load churn compared with the current scheduler.

Confirmations
  • I searched existing issues for duplicates.
  • I agree to follow the Code of Conduct.
Linguagem predominante
Go
Estrelas
1.6k
Forks
266
Merge médio
3d 6h
PRs com merge (30d)
22

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de NVIDIA/Personal-AI-Router

Todas as issues de NVIDIA/Personal-AI-Router

Issues semelhantes

Mais issues de Go

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.