SentencePieceTokenizer (Unigram) emits one <unk> per unknown character — SentencePiece merges a run of unknown characters into one <unk>

Ouverte
  • Quelque chose ne va pas ?
#7,859 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
58/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
csharp
Domaine
machine-learning

Piste de recherche

Start by locating SentencePieceTokenizer and its Unigram encoding path in the repository; compare how it handles consecutive unknown pieces with the SentencePiece behavior described in the issue. Check whether the BPE path shares the same behavior. Done means consecutive unknown pieces encode as one <unk> for Unigram models, matching the reference output on the repro cases.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area-Tokenizers untriaged

Summary

SentencePieceTokenizer.Create(stream) on a SentencePiece Unigram model returns one <unk> id for every character the vocabulary lacks. The reference implementation (google/sentencepiece unigram_model.cc, "merges continuous run of unknown pieces") and HF tokenizers return a single <unk> for a run of such characters. A model trained with SentencePiece therefore receives different ids from .NET than it was trained on.

Repro (Microsoft.ML.Tokenizers 2.0.1, net10.0, osx-arm64)

  1. Train a small Unigram model on a few Korean/English sentences that contain no Latin capitals: 
    spm.SentencePieceTrainer.train(input=corpus, model_type="unigram", vocab_size=80,
    normalization_rule_name="nmt_nfkc", unk_id=0, bos_id=1, eos_id=2, pad_id=-1)
  2. Encode "ＡＢＣ 전각 ①" (NFKC → ABC …, three characters absent from the vocabulary):
    • sentencepiece (Python), HF XLMRobertaTokenizer and XLMRobertaTokenizerFast: ▁ <unk> ▁ <unk> ▁ <unk> — one <unk> per run.
    • SentencePieceTokenizer.EncodeToIds(text, addBeginningOfSentence: false, addEndOfSentence: false): ▁ <unk> <unk> <unk> ▁ <unk> <unk> ▁ <unk> <unk> — one per character.
  3. Same for "처음 보는 글자 魑魅 😀": each unseen word becomes N <unk> instead of 1.

With a large vocabulary (XLM-RoBERTa's 250k sentencepiece.bpe.model) the two paths agreed on 12 everyday texts, because unknown runs are rare there — the difference shows on rare scripts and characters.

Expected

EncodeToIds merges consecutive unknown pieces into one <unk>, as SentencePiece's Unigram encoder does. (The BPE model path may be worth checking for the same behaviour.)

Workaround

Collapsing consecutive UnknownId entries after EncodeToIds. With that, a sentence encoder's vectors match sentence-transformers to cosine 1.000000 on 14 texts.

Langage dominant
C#
Étoiles
9.4k
Forks
2k
Merge moyen
2 j 18 h
PR mergées (30 j)
64

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dotnet/machinelearning

Toutes les issues de dotnet/machinelearning

Issues similaires

Plus d'issues C#

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.