Improve ONNX inference performance by adopting the OrtValue API

Aperta
  • Qualcosa non va?
#7,682 1 commento 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@vejbomar ci sta già lavorando.

Dal 22/9/2026.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
38/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
machine-learning, performance

Direzione di ricerca

Inizia leggendo il percorso di inferenza di OnnxTransformer ed eseguendo i test esistenti del trasformatore ONNX. Esegui un benchmark confrontando il percorso attuale NamedOnnxValue con un percorso OrtValue usando ONNX Runtime 1.23.2 nei casi di input e output elencati. Il lavoro è completo quando sono disponibili risultati delle prestazioni documentati, il comportamento supportato è preservato e non ci sono perdite di memoria nativa né buffer pinned persistenti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area-ONNX tenet-performance yellow

OnnxTransformer still invokes ONNX Runtime through the legacy NamedOnnxValue API. ONNX Runtime now recommends its OrtValue API, which produces less garbage, supports reusable input and output buffers, and can improve inference performance.

ML.NET currently references ONNX Runtime 1.23.2. The required C# API was introduced in 1.16, so upgrading ONNX Runtime is not a prerequisite for this work.

Motivation

The current inference path:

  • Creates NamedOnnxValue inputs for every row.
  • Copies numeric inputs using data.ToArray().
  • Creates intermediate string[] and DenseTensor<string> instances for string inputs.
  • Lets ONNX Runtime allocate output values for every invocation.
  • Converts and copies results into ML.NET VBuffer instances.
  • Performs one ONNX Runtime invocation per IDataView row.

These allocations can increase GC pressure and latency, particularly for small models and variable-length string inputs. This is related to #6620.

ONNX Runtime describes NamedOnnxValue as a legacy API and reports that OrtValue can provide substantial performance and allocation improvements in some scenarios.

Proposal

Benchmark and, where beneficial, migrate OnnxTransformer to the OrtValue API:

  1. Use OrtValue.CreateTensorValueFromMemory with reusable managed buffers for numeric inputs.
  2. Preallocate and reuse numeric output tensors when their shapes are fixed.
  3. Access dynamically allocated outputs through GetTensorDataAsSpan<T>().
  4. Populate native string tensors directly from ML.NET values, avoiding intermediate string[] and DenseTensor<string> allocations where possible.
  5. Reuse input names, output names, shapes, and RunOptions rather than recreating them for every row.
  6. Dispose native values deterministically and ensure managed buffers are not left pinned.
  7. Preserve the existing behaviour for dynamic shapes, strings, maps, sequences, and other supported ONNX value types.

String tensors will still require UTF-16 to UTF-8 conversion and copying, but the new API should allow us to remove several intermediate managed allocations.

Benchmark plan

Compare the existing and OrtValue paths using ONNX Runtime 1.23.2 first, so the effect of the API migration can be measured independently.

Cover at least:

  • Fixed-shape numeric inputs and outputs.
  • Dynamic-shape numeric inputs or outputs.
  • Fixed and variable-length string inputs.
  • Single and multiple input/output models.
  • Small models where managed overhead is significant.
  • Longer-running models to check for regressions.

Measure:

  • Mean inference latency and throughput.
  • Allocated bytes per inference.
  • Gen 0, Gen 1, and Gen 2 collections.
  • Performance under repeated scoring.
  • Memory and handle stability over prolonged execution.

The latest stable ONNX Runtime version should be evaluated separately. Any dependency update should ideally be made in a separate change so its kernel and execution-provider improvements can be measured independently.

Acceptance criteria

  • Benchmarks document the performance and allocation differences.
  • The OrtValue path is used where it improves performance without breaking existing behaviour.
  • Reusable buffers are used for fixed-shape numeric tensors.
  • Intermediate string arrays and tensors are removed where the API permits.
  • Existing ONNX transformer tests continue to pass.
  • Dynamic shapes and all currently supported input and output types remain compatible.
  • Repeated inference does not leak native memory or leave managed buffers pinned.

Out of scope

  • Changing IDataView to perform automatic batching.
  • Introducing device-backed VBuffer values.
  • Keeping complete pipelines resident in GPU memory.

Those changes may provide larger GPU throughput improvements, but require broader ML.NET API and pipeline design work. OrtIoBinding already has an official C# binding and can be investigated separately for such scenarios.

References

Lingua principale
C#
Stelle
9.4k
Fork
2k
Merge medio
2g 18h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dotnet/machinelearning

Tutte le issue di dotnet/machinelearning

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.