BertTokenizer: `RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under invariant globalization

Aperta
  • Qualcosa non va?
#7,728 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
machine-learning

Direzione di ricerca

Start at BertTokenizer.Create and the RemoveNonSpacingMarks handling, then run the supplied reproduction once with ICU and once with InvariantGlobalization=true. The issue is done when precomposed accents are handled consistently across both modes, or creation fails clearly when decomposition is unavailable; add coverage for the reported inputs and configuration.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area-Tokenizers green

BertOptions.RemoveNonSpacingMarks = true is the setting that makes BertTokenizer strip accents, as uncased BERT vocabularies expect: café becomes cafe. Accent stripping works by decomposing the text to Unicode Form D and dropping the non-spacing marks. Under globalization-invariant mode, string.Normalize returns its input unchanged — that is documented behaviour — so there are no marks to drop. Precomposed accented characters remain intact; for vocabularies such as all-MiniLM-L6-v2, this can cause affected words to miss their expected WordPiece decomposition and become [UNK].

Nothing signals this. BertTokenizer.Create succeeds, encoding succeeds, and for precomposed text the output is identical to having left the option off.

Invariant globalization is common in size-optimised .NET deployments. Microsoft's Alpine and Ubuntu Chiseled images ship without ICU (their extra variants add it), and the .NET 10 Native AOT templates enable it: dotnet new console --aot, dotnet new worker --aot and dotnet new webapiaot all set <InvariantGlobalization>true</InvariantGlobalization>. A service that tokenizes correctly in development can silently stop stripping accents when deployed in one of these configurations.

Reproduction

The reproduction below uses the vocab.txt from sentence-transformers/all-MiniLM-L6-v2.

using System.Text;
using Microsoft.ML.Tokenizers;

var vocabPath = args[0];
var vocab = File.ReadAllLines(vocabPath);
var tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

Console.WriteLine($"Normalize(FormD) of \"\\u00e9\": {"\u00e9".Normalize(NormalizationForm.FormD).Length} code unit(s)");
var ids = tokenizer.EncodeToIds("Café crème brûlée in São Paulo, naïve résumé");
Console.WriteLine(string.Join(' ', ids.Select(id => vocab[id])));

Run it once as built, and once in invariant mode:

dotnet run -- path/to/vocab.txt
dotnet run -p:InvariantGlobalization=true -- path/to/vocab.txt

Under ICU:

Normalize(FormD) of "\u00e9": 2 code unit(s)
[CLS] cafe cr ##eme br ##ule ##e in sao paulo , naive resume [SEP]

Under invariant globalization:

Normalize(FormD) of "\u00e9": 1 code unit(s)
[CLS] [UNK] [UNK] [UNK] in [UNK] paulo , [UNK] [UNK] [SEP]

The ICU output matches Hugging Face's BertTokenizer exactly. The invariant output is identical to running with RemoveNonSpacingMarks = false.

Only precomposed input is affected; input that is already decomposed is handled correctly because its combining marks are present to be dropped.

Expected behaviour

Either of these would remove the silent failure:

  1. Strip accents without relying on string.Normalize, for example by carrying the decomposition data the option needs, so it behaves the same in every globalization mode; or
  2. Fail loudly: throw from BertTokenizer.Create when RemoveNonSpacingMarks = true and decomposition is unavailable, with a message naming the fix (<InvariantGlobalization>false</InvariantGlobalization>, or an ICU-carrying image such as a chiseled -extra variant).

The check for (2) is one line at construction: "\u00e9".Normalize(NormalizationForm.FormD).Length == 2.

Impact

In my test with all-MiniLM-L6-v2, mean pooling, and cosine similarity against the ICU tokenization, the embeddings of affected text diverge substantially:

input ICU invariant
Café crème brûlée in São Paulo, naïve résumé 1.000000 0.333027
a French sentence 1.000000 0.545866
a Spanish sentence 1.000000 0.502484
a German sentence 1.000000 0.572320
accented capitals (ÉCOLE ÉLÉMENTAIRE) 1.000000 0.089375
plain ASCII 1.000000 1.000000
Versions
  • Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, identical output
  • .NET 10.0.12
Related

#7724 and #7725 report other BertTokenizer differences from Hugging Face's tokenizer. This one is independent of them: the output is correct under ICU and wrong only because of the globalization mode.

Lingua principale
C#
Stelle
9.4k
Fork
2k
Merge medio
2g 18h
PR unite (30g)
64

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di dotnet/machinelearning

Tutte le issue di dotnet/machinelearning

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.