SentencePieceTokenizer: charsmap entries that map to an empty string are not applied, so control characters become <unk>

Aperta
  • Qualcosa non va?
#7,908 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
69/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
machine-learning

Direzione di ricerca

Start by locating SentencePieceTokenizer normalization and the charsmap handling used by both CreateFromTokenizerJson and the .model protobuf path. Reproduce the reported case with a\u0001b and compare the normalized output with SentencePiece; done means all listed code points mapped to an empty string are deleted and token IDs match the expected behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404

Describe the bug
The precompiled nmt_nfkc charsmap used by SentencePiece Unigram models maps 30 control characters to an empty string, i.e. the normalizer deletes them. SentencePieceTokenizer does not apply these entries: the character survives normalization and is encoded as <unk>. As a result, token IDs differ from both Hugging Face tokenizers and SentencePiece C++ on any text that contains these characters.

The affected code points are U+0001–U+0008, U+000B, U+000E–U+001F, U+007F, U+008F and U+009F.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("a\u0001b")));
Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("hello\u007Fworld")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"a\u0001b" [10, 3, 275] (▁a, <unk>, b) [1563] (▁ab)
"hello\u007Fworld" [33600, 31, 3, 32554] [33600, 31, 32554]

Expected behavior
Charsmap entries whose replacement is an empty string should delete the matched input, as SentencePiece and Hugging Face do.

Additional context
We ran every Unicode scalar value (1,112,063 code points) through the normalizer and compared the result with SentencePiece's own normalization. 1,112,033 match; all 30 differences are the code points listed above. Removing these characters before calling the tokenizer brings the result to 1,112,063 of 1,112,063. The behavior is the same whether the tokenizer is created from tokenizer.json or from the .model protobuf.

Workaround: strip the 30 code points before encoding.

A related tie-breaking difference is reported in #7909.

Lingua principale
C#
Stelle
9.4k
Fork
2k
Merge medio
2g 18h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dotnet/machinelearning

Tutte le issue di dotnet/machinelearning

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.