Add reusable inference orchestration and dynamic batching

Abierto
  • ¿Algo no está bien?
#7,737 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
30/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
csharp
Área
ai-infra-agents, backend-api-design

Línea de trabajo

Look at existing model hosting and inference code in the repository, particularly around MLContext and prediction engines. Review the linked issues #7383 and microsoft/semantic-kernel#9793 for context. Start by exploring how batches are currently formed and executed, then design a scheduler interface that can be integrated with the existing prediction pipeline.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-Meta enhancement untriaged

Is your feature request related to a problem? Please describe.
Efficiently hosting custom AI models requires more than invoking a model for one request at a time. Applications may need to collect requests into bounded time windows, create batches, route work across model instances or devices, limit concurrency, and account for available accelerator memory. Teams that cannot use a separate inference server currently have to build and operate this orchestration themselves.

This is difficult to implement reliably and can prevent applications from fully utilizing GPUs while meeting latency targets.

Describe the solution you'd like
Investigate and provide reusable inference-orchestration building blocks for in-process .NET model hosting. The scope should consider:

  • Dynamic batching with configurable maximum batch size and wait time.
  • Backpressure, cancellation, queue limits, and per-request result correlation.
  • Concurrency control and routing across model sessions, execution providers, or devices.
  • Model placement and admission decisions informed by available memory or configured resource budgets.
  • Metrics for queue time, batch composition, execution time, utilization, and rejected work.
  • Extensible model-runner contracts so orchestration is not tied to a single inference backend.

A first increment could focus on a bounded dynamic-batching scheduler and leave memory-aware multi-model placement as a later extension.

Describe alternatives you've considered
Applications can build this with channels, queues, semaphores, and backend-specific session pools, or deploy a separate inference server such as NVIDIA Triton. Those approaches may be unsuitable when inference must remain in-process, model calls are interleaved with application logic, or deployment constraints prohibit an additional service.

Additional context
This request was identified in the “Bonus” section of the discussion that became #7383, originating in microsoft/semantic-kernel#9793. It is distinct from tokenizer batching: this issue concerns scheduling and executing inference requests after application inputs are ready for model processing.

[!NOTE]
This issue was drafted with AI assistance.

Lenguaje dominante
C#
Estrellas
9.4k
Forks
2k
Merge medio
2 d 18 h
PR fusionados (30 d)
64

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/machinelearning

Todos los issues de dotnet/machinelearning

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.