SentencePieceTokenizer (Unigram, from tokenizer.json): equal-score segmentations are ordered differently from Hugging Face tokenizers

Ouverte
  • Quelque chose ne va pas ?
#7,909 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
csharp
Domaine
machine-learning

Piste de recherche

Start with SentencePieceTokenizer.CreateFromTokenizerJson and the Unigram segmentation path; compare its equal-score tie handling with Hugging Face tokenizers 0.23.3's Unigram::encode_optimized behavior described in the issue. Reproduce with the multilingual-e5-small tokenizer.json and input "aaaaaaaaaaaa". Done when tokenizer.json encoding matches the expected Hugging Face token ID order for tie cases; the issue names no test file.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404
  • Reference: Hugging Face tokenizers 0.23.3

Describe the bug
On repetitive input, several Unigram segmentations can have exactly the same total score: they use the same pieces in a different order. When this happens, SentencePieceTokenizer picks a different path than Hugging Face tokenizers. The resulting token IDs are a permutation of the expected ones.

Hugging Face (Unigram::encode_optimized) keeps the path it finds first when scores are equal. Its final piece therefore starts as early as possible. SentencePiece C++ also differs from Hugging Face here, because it accumulates scores in float. So this is a parity issue rather than a single "correct" answer. But a tokenizer created with CreateFromTokenizerJson is naturally expected to reproduce the Hugging Face output for that file.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("aaaaaaaaaaaa")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"aaaaaaaaaaaa" [10, 83151, 83151, 11] (▁a, aaaaa, aaaaa, a) [10, 11, 83151, 83151] (▁a, a, aaaaa, aaaaa)

Expected behavior
When the tokenizer is created from tokenizer.json, ties between equal-score segmentations are resolved the way Hugging Face resolves them. Alternatively, an option to choose that behavior.

Additional context
On a corpus of 12,600 real-world strings, 247 differ from Hugging Face. In every case it is the same pieces in a different order. Typical triggers are runs such as 0000…, ────, ==== and аааа. For an embedding model the order matters: cosine similarity between the embeddings of the two ID sequences drops to 0.97 in the worst case (median 0.998). That breaks parity tests that expect ≥ 0.9999.

We reimplemented the Hugging Face Viterbi pass over the library's normalized output in about 100 lines. With it, all 12,600 strings match.

A related normalization bug (control characters becoming <unk>) is reported in #7908.

Langage dominant
C#
Étoiles
9.4k
Forks
2k
Merge moyen
2 j 18 h
PR mergées (30 j)
64

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dotnet/machinelearning

Toutes les issues de dotnet/machinelearning

Issues similaires

Plus d'issues C#

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.