Add reusable inference orchestration and dynamic batching

未关闭
  • 哪里不对？
#7,737 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
30/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
csharp
领域
ai-infra-agents, backend-api-design

调研方向

Look at existing model hosting and inference code in the repository, particularly around MLContext and prediction engines. Review the linked issues #7383 and microsoft/semantic-kernel#9793 for context. Start by exploring how batches are currently formed and executed, then design a scheduler interface that can be integrated with the existing prediction pipeline.

由索引模型根据 Issue 内容生成。

描述

area-Meta enhancement untriaged

Is your feature request related to a problem? Please describe.
Efficiently hosting custom AI models requires more than invoking a model for one request at a time. Applications may need to collect requests into bounded time windows, create batches, route work across model instances or devices, limit concurrency, and account for available accelerator memory. Teams that cannot use a separate inference server currently have to build and operate this orchestration themselves.

This is difficult to implement reliably and can prevent applications from fully utilizing GPUs while meeting latency targets.

Describe the solution you'd like
Investigate and provide reusable inference-orchestration building blocks for in-process .NET model hosting. The scope should consider:

  • Dynamic batching with configurable maximum batch size and wait time.
  • Backpressure, cancellation, queue limits, and per-request result correlation.
  • Concurrency control and routing across model sessions, execution providers, or devices.
  • Model placement and admission decisions informed by available memory or configured resource budgets.
  • Metrics for queue time, batch composition, execution time, utilization, and rejected work.
  • Extensible model-runner contracts so orchestration is not tied to a single inference backend.

A first increment could focus on a bounded dynamic-batching scheduler and leave memory-aware multi-model placement as a later extension.

Describe alternatives you've considered
Applications can build this with channels, queues, semaphores, and backend-specific session pools, or deploy a separate inference server such as NVIDIA Triton. Those approaches may be unsuitable when inference must remain in-process, model calls are interleaved with application logic, or deployment constraints prohibit an additional service.

Additional context
This request was identified in the “Bonus” section of the discussion that became #7383, originating in microsoft/semantic-kernel#9793. It is distinct from tokenizer batching: this issue concerns scheduling and executing inference requests after application inputs are ready for model processing.

[!NOTE]
This issue was drafted with AI assistance.

主要语言
C#
星标
9.4k
派生
2k
平均合并
2 天 18 小时
30 天内合并 PR
64

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

dotnet/machinelearning 的其他 Issue

查看 dotnet/machinelearning 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。