SentencePieceTokenizer: charsmap entries that map to an empty string are not applied, so control characters become <unk>

未关闭
  • 哪里不对？
#7,908 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
69/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
csharp
领域
machine-learning

调研方向

Start by locating SentencePieceTokenizer normalization and the charsmap handling used by both CreateFromTokenizerJson and the .model protobuf path. Reproduce the reported case with a\u0001b and compare the normalized output with SentencePiece; done means all listed code points mapped to an empty string are deleted and token IDs match the expected behavior.

由索引模型根据 Issue 内容生成。

描述

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404

Describe the bug
The precompiled nmt_nfkc charsmap used by SentencePiece Unigram models maps 30 control characters to an empty string, i.e. the normalizer deletes them. SentencePieceTokenizer does not apply these entries: the character survives normalization and is encoded as <unk>. As a result, token IDs differ from both Hugging Face tokenizers and SentencePiece C++ on any text that contains these characters.

The affected code points are U+0001–U+0008, U+000B, U+000E–U+001F, U+007F, U+008F and U+009F.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("a\u0001b")));
Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("hello\u007Fworld")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"a\u0001b" [10, 3, 275] (▁a, <unk>, b) [1563] (▁ab)
"hello\u007Fworld" [33600, 31, 3, 32554] [33600, 31, 32554]

Expected behavior
Charsmap entries whose replacement is an empty string should delete the matched input, as SentencePiece and Hugging Face do.

Additional context
We ran every Unicode scalar value (1,112,063 code points) through the normalizer and compared the result with SentencePiece's own normalization. 1,112,033 match; all 30 differences are the code points listed above. Removing these characters before calling the tokenizer brings the result to 1,112,063 of 1,112,063. The behavior is the same whether the tokenizer is created from tokenizer.json or from the .model protobuf.

Workaround: strip the 30 code points before encoding.

A related tie-breaking difference is reported in #7909.

主要语言
C#
星标
9.4k
派生
2k
平均合并
2 天 18 小时
30 天内合并 PR
64

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

dotnet/machinelearning 的其他 Issue

查看 dotnet/machinelearning 的全部 Issue

相似的 Issue

更多 C# Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。