Improve ONNX inference performance by adopting the OrtValue API

オープン
  • 何か問題がありますか？
#7,682 コメント 1 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@vejbomar がすでに取り組んでいます。

2026年9月22日 から。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
38/100
issue の種類
リファクタリング
明瞭さ
おおむね明確
活発さ
活発
技術スタック
csharp
領域
machine-learning, performance

調査の方向性

まず OnnxTransformer の推論パスを読み、既存の ONNX transformer テストを実行します。列挙された入力および出力ケース全体で、ONNX Runtime 1.23.2 を使用して、現在の NamedOnnxValue パスと OrtValue パスをベンチマーク比較します。完了の条件は、パフォーマンス結果が文書化され、サポート対象の動作が維持され、ネイティブメモリリークや残存する pinned バッファーがないことです。

索引モデルが issue の本文から書いたものです。

説明

area-ONNX tenet-performance yellow

OnnxTransformer still invokes ONNX Runtime through the legacy NamedOnnxValue API. ONNX Runtime now recommends its OrtValue API, which produces less garbage, supports reusable input and output buffers, and can improve inference performance.

ML.NET currently references ONNX Runtime 1.23.2. The required C# API was introduced in 1.16, so upgrading ONNX Runtime is not a prerequisite for this work.

Motivation

The current inference path:

  • Creates NamedOnnxValue inputs for every row.
  • Copies numeric inputs using data.ToArray().
  • Creates intermediate string[] and DenseTensor<string> instances for string inputs.
  • Lets ONNX Runtime allocate output values for every invocation.
  • Converts and copies results into ML.NET VBuffer instances.
  • Performs one ONNX Runtime invocation per IDataView row.

These allocations can increase GC pressure and latency, particularly for small models and variable-length string inputs. This is related to #6620.

ONNX Runtime describes NamedOnnxValue as a legacy API and reports that OrtValue can provide substantial performance and allocation improvements in some scenarios.

Proposal

Benchmark and, where beneficial, migrate OnnxTransformer to the OrtValue API:

  1. Use OrtValue.CreateTensorValueFromMemory with reusable managed buffers for numeric inputs.
  2. Preallocate and reuse numeric output tensors when their shapes are fixed.
  3. Access dynamically allocated outputs through GetTensorDataAsSpan<T>().
  4. Populate native string tensors directly from ML.NET values, avoiding intermediate string[] and DenseTensor<string> allocations where possible.
  5. Reuse input names, output names, shapes, and RunOptions rather than recreating them for every row.
  6. Dispose native values deterministically and ensure managed buffers are not left pinned.
  7. Preserve the existing behaviour for dynamic shapes, strings, maps, sequences, and other supported ONNX value types.

String tensors will still require UTF-16 to UTF-8 conversion and copying, but the new API should allow us to remove several intermediate managed allocations.

Benchmark plan

Compare the existing and OrtValue paths using ONNX Runtime 1.23.2 first, so the effect of the API migration can be measured independently.

Cover at least:

  • Fixed-shape numeric inputs and outputs.
  • Dynamic-shape numeric inputs or outputs.
  • Fixed and variable-length string inputs.
  • Single and multiple input/output models.
  • Small models where managed overhead is significant.
  • Longer-running models to check for regressions.

Measure:

  • Mean inference latency and throughput.
  • Allocated bytes per inference.
  • Gen 0, Gen 1, and Gen 2 collections.
  • Performance under repeated scoring.
  • Memory and handle stability over prolonged execution.

The latest stable ONNX Runtime version should be evaluated separately. Any dependency update should ideally be made in a separate change so its kernel and execution-provider improvements can be measured independently.

Acceptance criteria

  • Benchmarks document the performance and allocation differences.
  • The OrtValue path is used where it improves performance without breaking existing behaviour.
  • Reusable buffers are used for fixed-shape numeric tensors.
  • Intermediate string arrays and tensors are removed where the API permits.
  • Existing ONNX transformer tests continue to pass.
  • Dynamic shapes and all currently supported input and output types remain compatible.
  • Repeated inference does not leak native memory or leave managed buffers pinned.

Out of scope

  • Changing IDataView to perform automatic batching.
  • Introducing device-backed VBuffer values.
  • Keeping complete pipelines resident in GPU memory.

Those changes may provide larger GPU throughput improvements, but require broader ML.NET API and pipeline design work. OrtIoBinding already has an official C# binding and can be investigated separately for such scenarios.

References

主要言語
C#
スター
9.4k
フォーク
2k
平均マージ
2日 18時間
マージ済み PR（30日）
64

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dotnet/machinelearning のほかの issue

dotnet/machinelearning の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。