SentencePieceTokenizer (Unigram, from tokenizer.json): equal-score segmentations are ordered differently from Hugging Face tokenizers

未關閉
  • 哪裡不對？
#7,909 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
4/5
預估耗時
3-5 天
新手友好度
48/100
Issue 類型
缺陷
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
csharp
領域
machine-learning

研究方向

Start with SentencePieceTokenizer.CreateFromTokenizerJson and the Unigram segmentation path; compare its equal-score tie handling with Hugging Face tokenizers 0.23.3's Unigram::encode_optimized behavior described in the issue. Reproduce with the multilingual-e5-small tokenizer.json and input "aaaaaaaaaaaa". Done when tokenizer.json encoding matches the expected Hugging Face token ID order for tie cases; the issue names no test file.

由索引模型根據 Issue 內容生成。

描述

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404
  • Reference: Hugging Face tokenizers 0.23.3

Describe the bug
On repetitive input, several Unigram segmentations can have exactly the same total score: they use the same pieces in a different order. When this happens, SentencePieceTokenizer picks a different path than Hugging Face tokenizers. The resulting token IDs are a permutation of the expected ones.

Hugging Face (Unigram::encode_optimized) keeps the path it finds first when scores are equal. Its final piece therefore starts as early as possible. SentencePiece C++ also differs from Hugging Face here, because it accumulates scores in float. So this is a parity issue rather than a single "correct" answer. But a tokenizer created with CreateFromTokenizerJson is naturally expected to reproduce the Hugging Face output for that file.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("aaaaaaaaaaaa")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"aaaaaaaaaaaa" [10, 83151, 83151, 11] (▁a, aaaaa, aaaaa, a) [10, 11, 83151, 83151] (▁a, a, aaaaa, aaaaa)

Expected behavior
When the tokenizer is created from tokenizer.json, ties between equal-score segmentations are resolved the way Hugging Face resolves them. Alternatively, an option to choose that behavior.

Additional context
On a corpus of 12,600 real-world strings, 247 differ from Hugging Face. In every case it is the same pieces in a different order. Typical triggers are runs such as 0000…, ────, ==== and аааа. For an embedding model the order matters: cosine similarity between the embeddings of the two ID sequences drops to 0.97 in the worst case (median 0.998). That breaks parity tests that expect ≥ 0.9999.

We reimplemented the Hugging Face Viterbi pass over the library's normalized output in about 100 lines. With it, all 12,600 strings match.

A related normalization bug (control characters becoming <unk>) is reported in #7908.

主要語言
C#
星號
9.4k
分支
2k
平均合併
2 天 18 小時
30 天內合併 PR
64

環境準備

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

dotnet/machinelearning 的其他 Issue

查看 dotnet/machinelearning 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。