BertTokenizer: `RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under invariant globalization

Ouverte
  • Quelque chose ne va pas ?
#7,728 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
csharp
Domaine
machine-learning

Piste de recherche

Start at BertTokenizer.Create and the RemoveNonSpacingMarks handling, then run the supplied reproduction once with ICU and once with InvariantGlobalization=true. The issue is done when precomposed accents are handled consistently across both modes, or creation fails clearly when decomposition is unavailable; add coverage for the reported inputs and configuration.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

area-Tokenizers green

BertOptions.RemoveNonSpacingMarks = true is the setting that makes BertTokenizer strip accents, as uncased BERT vocabularies expect: café becomes cafe. Accent stripping works by decomposing the text to Unicode Form D and dropping the non-spacing marks. Under globalization-invariant mode, string.Normalize returns its input unchanged — that is documented behaviour — so there are no marks to drop. Precomposed accented characters remain intact; for vocabularies such as all-MiniLM-L6-v2, this can cause affected words to miss their expected WordPiece decomposition and become [UNK].

Nothing signals this. BertTokenizer.Create succeeds, encoding succeeds, and for precomposed text the output is identical to having left the option off.

Invariant globalization is common in size-optimised .NET deployments. Microsoft's Alpine and Ubuntu Chiseled images ship without ICU (their extra variants add it), and the .NET 10 Native AOT templates enable it: dotnet new console --aot, dotnet new worker --aot and dotnet new webapiaot all set <InvariantGlobalization>true</InvariantGlobalization>. A service that tokenizes correctly in development can silently stop stripping accents when deployed in one of these configurations.

Reproduction

The reproduction below uses the vocab.txt from sentence-transformers/all-MiniLM-L6-v2.

using System.Text;
using Microsoft.ML.Tokenizers;

var vocabPath = args[0];
var vocab = File.ReadAllLines(vocabPath);
var tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

Console.WriteLine($"Normalize(FormD) of \"\\u00e9\": {"\u00e9".Normalize(NormalizationForm.FormD).Length} code unit(s)");
var ids = tokenizer.EncodeToIds("Café crème brûlée in São Paulo, naïve résumé");
Console.WriteLine(string.Join(' ', ids.Select(id => vocab[id])));

Run it once as built, and once in invariant mode:

dotnet run -- path/to/vocab.txt
dotnet run -p:InvariantGlobalization=true -- path/to/vocab.txt

Under ICU:

Normalize(FormD) of "\u00e9": 2 code unit(s)
[CLS] cafe cr ##eme br ##ule ##e in sao paulo , naive resume [SEP]

Under invariant globalization:

Normalize(FormD) of "\u00e9": 1 code unit(s)
[CLS] [UNK] [UNK] [UNK] in [UNK] paulo , [UNK] [UNK] [SEP]

The ICU output matches Hugging Face's BertTokenizer exactly. The invariant output is identical to running with RemoveNonSpacingMarks = false.

Only precomposed input is affected; input that is already decomposed is handled correctly because its combining marks are present to be dropped.

Expected behaviour

Either of these would remove the silent failure:

  1. Strip accents without relying on string.Normalize, for example by carrying the decomposition data the option needs, so it behaves the same in every globalization mode; or
  2. Fail loudly: throw from BertTokenizer.Create when RemoveNonSpacingMarks = true and decomposition is unavailable, with a message naming the fix (<InvariantGlobalization>false</InvariantGlobalization>, or an ICU-carrying image such as a chiseled -extra variant).

The check for (2) is one line at construction: "\u00e9".Normalize(NormalizationForm.FormD).Length == 2.

Impact

In my test with all-MiniLM-L6-v2, mean pooling, and cosine similarity against the ICU tokenization, the embeddings of affected text diverge substantially:

input ICU invariant
Café crème brûlée in São Paulo, naïve résumé 1.000000 0.333027
a French sentence 1.000000 0.545866
a Spanish sentence 1.000000 0.502484
a German sentence 1.000000 0.572320
accented capitals (ÉCOLE ÉLÉMENTAIRE) 1.000000 0.089375
plain ASCII 1.000000 1.000000
Versions
  • Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, identical output
  • .NET 10.0.12
Related

#7724 and #7725 report other BertTokenizer differences from Hugging Face's tokenizer. This one is independent of them: the output is correct under ICU and wrong only because of the globalization mode.

Langage dominant
C#
Étoiles
9.4k
Forks
2k
Merge moyen
2 j 18 h
PR mergées (30 j)
64

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de dotnet/machinelearning

Toutes les issues de dotnet/machinelearning

Issues similaires

Plus d'issues C#

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.