BertTokenizer: `RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under invariant globalization

Offen
  • Stimmt etwas nicht?
#7,728 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
48/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
csharp
Bereich
machine-learning

Rechercherichtung

Start at BertTokenizer.Create and the RemoveNonSpacingMarks handling, then run the supplied reproduction once with ICU and once with InvariantGlobalization=true. The issue is done when precomposed accents are handled consistently across both modes, or creation fails clearly when decomposition is unavailable; add coverage for the reported inputs and configuration.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

area-Tokenizers green

BertOptions.RemoveNonSpacingMarks = true is the setting that makes BertTokenizer strip accents, as uncased BERT vocabularies expect: café becomes cafe. Accent stripping works by decomposing the text to Unicode Form D and dropping the non-spacing marks. Under globalization-invariant mode, string.Normalize returns its input unchanged — that is documented behaviour — so there are no marks to drop. Precomposed accented characters remain intact; for vocabularies such as all-MiniLM-L6-v2, this can cause affected words to miss their expected WordPiece decomposition and become [UNK].

Nothing signals this. BertTokenizer.Create succeeds, encoding succeeds, and for precomposed text the output is identical to having left the option off.

Invariant globalization is common in size-optimised .NET deployments. Microsoft's Alpine and Ubuntu Chiseled images ship without ICU (their extra variants add it), and the .NET 10 Native AOT templates enable it: dotnet new console --aot, dotnet new worker --aot and dotnet new webapiaot all set <InvariantGlobalization>true</InvariantGlobalization>. A service that tokenizes correctly in development can silently stop stripping accents when deployed in one of these configurations.

Reproduction

The reproduction below uses the vocab.txt from sentence-transformers/all-MiniLM-L6-v2.

using System.Text;
using Microsoft.ML.Tokenizers;

var vocabPath = args[0];
var vocab = File.ReadAllLines(vocabPath);
var tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

Console.WriteLine($"Normalize(FormD) of \"\\u00e9\": {"\u00e9".Normalize(NormalizationForm.FormD).Length} code unit(s)");
var ids = tokenizer.EncodeToIds("Café crème brûlée in São Paulo, naïve résumé");
Console.WriteLine(string.Join(' ', ids.Select(id => vocab[id])));

Run it once as built, and once in invariant mode:

dotnet run -- path/to/vocab.txt
dotnet run -p:InvariantGlobalization=true -- path/to/vocab.txt

Under ICU:

Normalize(FormD) of "\u00e9": 2 code unit(s)
[CLS] cafe cr ##eme br ##ule ##e in sao paulo , naive resume [SEP]

Under invariant globalization:

Normalize(FormD) of "\u00e9": 1 code unit(s)
[CLS] [UNK] [UNK] [UNK] in [UNK] paulo , [UNK] [UNK] [SEP]

The ICU output matches Hugging Face's BertTokenizer exactly. The invariant output is identical to running with RemoveNonSpacingMarks = false.

Only precomposed input is affected; input that is already decomposed is handled correctly because its combining marks are present to be dropped.

Expected behaviour

Either of these would remove the silent failure:

  1. Strip accents without relying on string.Normalize, for example by carrying the decomposition data the option needs, so it behaves the same in every globalization mode; or
  2. Fail loudly: throw from BertTokenizer.Create when RemoveNonSpacingMarks = true and decomposition is unavailable, with a message naming the fix (<InvariantGlobalization>false</InvariantGlobalization>, or an ICU-carrying image such as a chiseled -extra variant).

The check for (2) is one line at construction: "\u00e9".Normalize(NormalizationForm.FormD).Length == 2.

Impact

In my test with all-MiniLM-L6-v2, mean pooling, and cosine similarity against the ICU tokenization, the embeddings of affected text diverge substantially:

input ICU invariant
Café crème brûlée in São Paulo, naïve résumé 1.000000 0.333027
a French sentence 1.000000 0.545866
a Spanish sentence 1.000000 0.502484
a German sentence 1.000000 0.572320
accented capitals (ÉCOLE ÉLÉMENTAIRE) 1.000000 0.089375
plain ASCII 1.000000 1.000000
Versions
  • Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, identical output
  • .NET 10.0.12
Related

#7724 and #7725 report other BertTokenizer differences from Hugging Face's tokenizer. This one is independent of them: the output is correct under ICU and wrong only because of the globalization mode.

Vorherrschende Sprache
C#
Sterne
9.4k
Forks
2k
Ø Merge
2 T. 18 Std.
Gemergte PRs (30 T.)
64

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus dotnet/machinelearning

Alle Issues in dotnet/machinelearning

Ähnliche Issues

Weitere Issues zu C#

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.