SentencePieceTokenizer (Unigram) emits one <unk> per unknown character — SentencePiece merges a run of unknown characters into one <unk>

Abierto
  • ¿Algo no está bien?
#7,859 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
58/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
csharp
Área
machine-learning

Línea de trabajo

Start by locating SentencePieceTokenizer and its Unigram encoding path in the repository; compare how it handles consecutive unknown pieces with the SentencePiece behavior described in the issue. Check whether the BPE path shares the same behavior. Done means consecutive unknown pieces encode as one <unk> for Unigram models, matching the reference output on the repro cases.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-Tokenizers untriaged

Summary

SentencePieceTokenizer.Create(stream) on a SentencePiece Unigram model returns one <unk> id for every character the vocabulary lacks. The reference implementation (google/sentencepiece unigram_model.cc, "merges continuous run of unknown pieces") and HF tokenizers return a single <unk> for a run of such characters. A model trained with SentencePiece therefore receives different ids from .NET than it was trained on.

Repro (Microsoft.ML.Tokenizers 2.0.1, net10.0, osx-arm64)

  1. Train a small Unigram model on a few Korean/English sentences that contain no Latin capitals: 
    spm.SentencePieceTrainer.train(input=corpus, model_type="unigram", vocab_size=80,
    normalization_rule_name="nmt_nfkc", unk_id=0, bos_id=1, eos_id=2, pad_id=-1)
  2. Encode "ＡＢＣ 전각 ①" (NFKC → ABC …, three characters absent from the vocabulary):
    • sentencepiece (Python), HF XLMRobertaTokenizer and XLMRobertaTokenizerFast: ▁ <unk> ▁ <unk> ▁ <unk> — one <unk> per run.
    • SentencePieceTokenizer.EncodeToIds(text, addBeginningOfSentence: false, addEndOfSentence: false): ▁ <unk> <unk> <unk> ▁ <unk> <unk> ▁ <unk> <unk> — one per character.
  3. Same for "처음 보는 글자 魑魅 😀": each unseen word becomes N <unk> instead of 1.

With a large vocabulary (XLM-RoBERTa's 250k sentencepiece.bpe.model) the two paths agreed on 12 everyday texts, because unknown runs are rare there — the difference shows on rare scripts and characters.

Expected

EncodeToIds merges consecutive unknown pieces into one <unk>, as SentencePiece's Unigram encoder does. (The BPE model path may be worth checking for the same behaviour.)

Workaround

Collapsing consecutive UnknownId entries after EncodeToIds. With that, a sentence encoder's vectors match sentence-transformers to cosine 1.000000 on 14 texts.

Lenguaje dominante
C#
Estrellas
9.4k
Forks
2k
Merge medio
2 d 18 h
PR fusionados (30 d)
64

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/machinelearning

Todos los issues de dotnet/machinelearning

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.