ONNX export of a reloaded NgramExtractingTransformer (useAllLengths: true) writes min_gram_length = ngramLength — the exported graph counts only the longest n-grams

Ouverte
  • Quelque chose ne va pas ?
#7,824 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
72/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
csharp, machine-learning
Domaine
backend, machine-learning

Piste de recherche

Start by tracing ONNX export for NgramExtractingTransformer, comparing the export path immediately after Fit with the path after Model.Save and Model.Load. The issue provides a minimal C# reproduction and expected graph attributes: min_gram_length should be 1 with useAllLengths: true. Reproduce the value mismatch on the provided text output and confirm the reloaded export counts all n-gram lengths.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area-ONNX untriaged
System information
  • Microsoft.ML 5.0.1, Microsoft.ML.OnnxConverter 0.23.1, Microsoft.ML.OnnxRuntime 1.31.0
  • .NET 10 (net10.0), macOS arm64
Describe the bug

An NgramExtractingTransformer created with useAllLengths: true exports to ONNX correctly when it is converted straight after Fit. After a Model.Save / Model.Load round trip, the same transformer exports a TfIdfVectorizer node with min_gram_length equal to max_gram_length. The resulting graph counts only the longest n-grams. The conversion reports success, so nothing shows that the graph is wrong.

To reproduce

Text column of the UCI SMS Spam Collection (4,459 training rows, 1,115 compared):

var t = ml.Transforms.Text.TokenizeIntoCharactersAsKeys("c", "text")
    .Append(ml.Transforms.Text.ProduceNgrams("text", "c", ngramLength: 4, useAllLengths: true,
        weighting: NgramExtractingEstimator.WeightingCriteria.Tf))
    .Fit(train);

The text output vector (width 75,141) from ML.NET, compared with the same output of the exported graph run in ONNX Runtime:

rows that differ
t exported directly (ml.Model.ConvertToOnnx(t, train, stream, "text")) 0 / 1,115
ml.Model.Save(t, …) → ml.Model.Load(…) → export 1,115 / 1,115

In the graph of the reloaded transformer, the TfIdfVectorizer node has min_gram_length: 4, max_gram_length: 4 while ngram_counts is [0, 112, 6500, 59291]. The pool holds 1-, 2-, 3- and 4-grams, but only 4-grams are counted, so every 1–3-gram slot is 0. A logistic regression trained on these features scores every row of the reloaded export at about its intercept (probability ≈ 0.0017; ML.NET gives 0.0000–1.0000).

Workaround: use one ProduceNgrams per length (useAllLengths: false, lengths 1..4) followed by Concatenate. This exports correctly after a reload.

Expected behavior

The exported min_gram_length is 1 when useAllLengths is true, whether or not the transformer went through Model.Save / Model.Load. In other words, the all-lengths setting survives deserialization in what the ONNX save path reads.

Why it matters

Models are usually exported from a saved file (train → save → export later). In that path the conversion succeeds and the graph silently produces different values. Only a value-by-value comparison catches it.

Langage dominant
C#
Étoiles
9.4k
Forks
2k
Merge moyen
2 j 18 h
PR mergées (30 j)
64

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dotnet/machinelearning

Toutes les issues de dotnet/machinelearning

Issues similaires

Plus d'issues C#

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.