SentencePieceTokenizer (Unigram, from tokenizer.json): equal-score segmentations are ordered differently from Hugging Face tokenizers

Offen
  • Stimmt etwas nicht?
#7,909 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
csharp
Bereich
machine-learning

Rechercherichtung

Start with SentencePieceTokenizer.CreateFromTokenizerJson and the Unigram segmentation path; compare its equal-score tie handling with Hugging Face tokenizers 0.23.3's Unigram::encode_optimized behavior described in the issue. Reproduce with the multilingual-e5-small tokenizer.json and input "aaaaaaaaaaaa". Done when tokenizer.json encoding matches the expected Hugging Face token ID order for tie cases; the issue names no test file.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404
  • Reference: Hugging Face tokenizers 0.23.3

Describe the bug
On repetitive input, several Unigram segmentations can have exactly the same total score: they use the same pieces in a different order. When this happens, SentencePieceTokenizer picks a different path than Hugging Face tokenizers. The resulting token IDs are a permutation of the expected ones.

Hugging Face (Unigram::encode_optimized) keeps the path it finds first when scores are equal. Its final piece therefore starts as early as possible. SentencePiece C++ also differs from Hugging Face here, because it accumulates scores in float. So this is a parity issue rather than a single "correct" answer. But a tokenizer created with CreateFromTokenizerJson is naturally expected to reproduce the Hugging Face output for that file.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("aaaaaaaaaaaa")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"aaaaaaaaaaaa" [10, 83151, 83151, 11] (▁a, aaaaa, aaaaa, a) [10, 11, 83151, 83151] (▁a, a, aaaaa, aaaaa)

Expected behavior
When the tokenizer is created from tokenizer.json, ties between equal-score segmentations are resolved the way Hugging Face resolves them. Alternatively, an option to choose that behavior.

Additional context
On a corpus of 12,600 real-world strings, 247 differ from Hugging Face. In every case it is the same pieces in a different order. Typical triggers are runs such as 0000…, ────, ==== and аааа. For an embedding model the order matters: cosine similarity between the embeddings of the two ID sequences drops to 0.97 in the worst case (median 0.998). That breaks parity tests that expect ≥ 0.9999.

We reimplemented the Hugging Face Viterbi pass over the library's normalized output in about 100 lines. With it, all 12,600 strings match.

A related normalization bug (control characters becoming <unk>) is reported in #7908.

Vorherrschende Sprache
C#
Sterne
9.4k
Forks
2k
Ø Merge
2 T. 18 Std.
Gemergte PRs (30 T.)
64

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus dotnet/machinelearning

Alle Issues in dotnet/machinelearning

Ähnliche Issues

Weitere Issues zu C#

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.