Add reusable inference orchestration and dynamic batching

オープン
  • 何か問題がありますか？
#7,737 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
30/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
csharp
領域
ai-infra-agents, backend-api-design

調査の方向性

Look at existing model hosting and inference code in the repository, particularly around MLContext and prediction engines. Review the linked issues #7383 and microsoft/semantic-kernel#9793 for context. Start by exploring how batches are currently formed and executed, then design a scheduler interface that can be integrated with the existing prediction pipeline.

索引モデルが issue の本文から書いたものです。

説明

area-Meta enhancement untriaged

Is your feature request related to a problem? Please describe.
Efficiently hosting custom AI models requires more than invoking a model for one request at a time. Applications may need to collect requests into bounded time windows, create batches, route work across model instances or devices, limit concurrency, and account for available accelerator memory. Teams that cannot use a separate inference server currently have to build and operate this orchestration themselves.

This is difficult to implement reliably and can prevent applications from fully utilizing GPUs while meeting latency targets.

Describe the solution you'd like
Investigate and provide reusable inference-orchestration building blocks for in-process .NET model hosting. The scope should consider:

  • Dynamic batching with configurable maximum batch size and wait time.
  • Backpressure, cancellation, queue limits, and per-request result correlation.
  • Concurrency control and routing across model sessions, execution providers, or devices.
  • Model placement and admission decisions informed by available memory or configured resource budgets.
  • Metrics for queue time, batch composition, execution time, utilization, and rejected work.
  • Extensible model-runner contracts so orchestration is not tied to a single inference backend.

A first increment could focus on a bounded dynamic-batching scheduler and leave memory-aware multi-model placement as a later extension.

Describe alternatives you've considered
Applications can build this with channels, queues, semaphores, and backend-specific session pools, or deploy a separate inference server such as NVIDIA Triton. Those approaches may be unsuitable when inference must remain in-process, model calls are interleaved with application logic, or deployment constraints prohibit an additional service.

Additional context
This request was identified in the “Bonus” section of the discussion that became #7383, originating in microsoft/semantic-kernel#9793. It is distinct from tokenizer batching: this issue concerns scheduling and executing inference requests after application inputs are ready for model processing.

[!NOTE]
This issue was drafted with AI assistance.

主要言語
C#
スター
9.4k
フォーク
2k
平均マージ
2日 18時間
マージ済み PR（30日）
64

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dotnet/machinelearning のほかの issue

dotnet/machinelearning の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。