SentencePieceTokenizer (Unigram, from tokenizer.json): equal-score segmentations are ordered differently from Hugging Face tokenizers

Aberta
  • Algo errado?
#7,909 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
csharp
Domínio
machine-learning

Direção de pesquisa

Start with SentencePieceTokenizer.CreateFromTokenizerJson and the Unigram segmentation path; compare its equal-score tie handling with Hugging Face tokenizers 0.23.3's Unigram::encode_optimized behavior described in the issue. Reproduce with the multilingual-e5-small tokenizer.json and input "aaaaaaaaaaaa". Done when tokenizer.json encoding matches the expected Hugging Face token ID order for tie cases; the issue names no test file.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

area-Tokenizers untriaged

System Information:

  • OS & Version: Debian 13 (trixie), x64
  • ML.NET Version: Microsoft.ML.Tokenizers 3.0.0
  • .NET Version: .NET SDK 10.0.404
  • Reference: Hugging Face tokenizers 0.23.3

Describe the bug
On repetitive input, several Unigram segmentations can have exactly the same total score: they use the same pieces in a different order. When this happens, SentencePieceTokenizer picks a different path than Hugging Face tokenizers. The resulting token IDs are a permutation of the expected ones.

Hugging Face (Unigram::encode_optimized) keeps the path it finds first when scores are equal. Its final piece therefore starts as early as possible. SentencePiece C++ also differs from Hugging Face here, because it accumulates scores in float. So this is a parity issue rather than a single "correct" answer. But a tokenizer created with CreateFromTokenizerJson is naturally expected to reproduce the Hugging Face output for that file.

To Reproduce
Model: intfloat/multilingual-e5-small, revision 614241f622f53c4eeff9890bdc4f31cfecc418b3, file onnx/tokenizer.json.

using Microsoft.ML.Tokenizers;

using var stream = File.OpenRead("tokenizer.json");
var tokenizer = SentencePieceTokenizer.CreateFromTokenizerJson(
    stream, addBeginningOfSentence: false, addEndOfSentence: false);

Console.WriteLine(string.Join(", ", tokenizer.EncodeToIds("aaaaaaaaaaaa")));
Input Microsoft.ML.Tokenizers 3.0.0 Hugging Face tokenizers 0.23.3
"aaaaaaaaaaaa" [10, 83151, 83151, 11] (▁a, aaaaa, aaaaa, a) [10, 11, 83151, 83151] (▁a, a, aaaaa, aaaaa)

Expected behavior
When the tokenizer is created from tokenizer.json, ties between equal-score segmentations are resolved the way Hugging Face resolves them. Alternatively, an option to choose that behavior.

Additional context
On a corpus of 12,600 real-world strings, 247 differ from Hugging Face. In every case it is the same pieces in a different order. Typical triggers are runs such as 0000…, ────, ==== and аааа. For an embedding model the order matters: cosine similarity between the embeddings of the two ID sequences drops to 0.97 in the worst case (median 0.998). That breaks parity tests that expect ≥ 0.9999.

We reimplemented the Hugging Face Viterbi pass over the library's normalized output in about 100 lines. With it, all 12,600 strings match.

A related normalization bug (control characters becoming <unk>) is reported in #7908.

Linguagem predominante
C#
Estrelas
9.4k
Forks
2k
Merge médio
2d 18h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dotnet/machinelearning

Todas as issues de dotnet/machinelearning

Issues semelhantes

Mais issues de C#

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.