ONNX export of a reloaded NgramExtractingTransformer (useAllLengths: true) writes min_gram_length = ngramLength — the exported graph counts only the longest n-grams

オープン
  • 何か問題がありますか？
#7,824 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
72/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
csharp, machine-learning
領域
backend, machine-learning

調査の方向性

Start by tracing ONNX export for NgramExtractingTransformer, comparing the export path immediately after Fit with the path after Model.Save and Model.Load. The issue provides a minimal C# reproduction and expected graph attributes: min_gram_length should be 1 with useAllLengths: true. Reproduce the value mismatch on the provided text output and confirm the reloaded export counts all n-gram lengths.

索引モデルが issue の本文から書いたものです。

説明

area-ONNX untriaged
System information
  • Microsoft.ML 5.0.1, Microsoft.ML.OnnxConverter 0.23.1, Microsoft.ML.OnnxRuntime 1.31.0
  • .NET 10 (net10.0), macOS arm64
Describe the bug

An NgramExtractingTransformer created with useAllLengths: true exports to ONNX correctly when it is converted straight after Fit. After a Model.Save / Model.Load round trip, the same transformer exports a TfIdfVectorizer node with min_gram_length equal to max_gram_length. The resulting graph counts only the longest n-grams. The conversion reports success, so nothing shows that the graph is wrong.

To reproduce

Text column of the UCI SMS Spam Collection (4,459 training rows, 1,115 compared):

var t = ml.Transforms.Text.TokenizeIntoCharactersAsKeys("c", "text")
    .Append(ml.Transforms.Text.ProduceNgrams("text", "c", ngramLength: 4, useAllLengths: true,
        weighting: NgramExtractingEstimator.WeightingCriteria.Tf))
    .Fit(train);

The text output vector (width 75,141) from ML.NET, compared with the same output of the exported graph run in ONNX Runtime:

rows that differ
t exported directly (ml.Model.ConvertToOnnx(t, train, stream, "text")) 0 / 1,115
ml.Model.Save(t, …) → ml.Model.Load(…) → export 1,115 / 1,115

In the graph of the reloaded transformer, the TfIdfVectorizer node has min_gram_length: 4, max_gram_length: 4 while ngram_counts is [0, 112, 6500, 59291]. The pool holds 1-, 2-, 3- and 4-grams, but only 4-grams are counted, so every 1–3-gram slot is 0. A logistic regression trained on these features scores every row of the reloaded export at about its intercept (probability ≈ 0.0017; ML.NET gives 0.0000–1.0000).

Workaround: use one ProduceNgrams per length (useAllLengths: false, lengths 1..4) followed by Concatenate. This exports correctly after a reload.

Expected behavior

The exported min_gram_length is 1 when useAllLengths is true, whether or not the transformer went through Model.Save / Model.Load. In other words, the all-lengths setting survives deserialization in what the ONNX save path reads.

Why it matters

Models are usually exported from a saved file (train → save → export later). In that path the conversion succeeds and the graph silently produces different values. Only a value-by-value comparison catches it.

主要言語
C#
スター
9.4k
フォーク
2k
平均マージ
2日 18時間
マージ済み PR（30日）
64

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dotnet/machinelearning のほかの issue

dotnet/machinelearning の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。