BertTokenizer: `RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under invariant globalization

Abierto
  • ¿Algo no está bien?
#7,728 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
csharp
Área
machine-learning

Línea de trabajo

Start at BertTokenizer.Create and the RemoveNonSpacingMarks handling, then run the supplied reproduction once with ICU and once with InvariantGlobalization=true. The issue is done when precomposed accents are handled consistently across both modes, or creation fails clearly when decomposition is unavailable; add coverage for the reported inputs and configuration.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area-Tokenizers green

BertOptions.RemoveNonSpacingMarks = true is the setting that makes BertTokenizer strip accents, as uncased BERT vocabularies expect: café becomes cafe. Accent stripping works by decomposing the text to Unicode Form D and dropping the non-spacing marks. Under globalization-invariant mode, string.Normalize returns its input unchanged — that is documented behaviour — so there are no marks to drop. Precomposed accented characters remain intact; for vocabularies such as all-MiniLM-L6-v2, this can cause affected words to miss their expected WordPiece decomposition and become [UNK].

Nothing signals this. BertTokenizer.Create succeeds, encoding succeeds, and for precomposed text the output is identical to having left the option off.

Invariant globalization is common in size-optimised .NET deployments. Microsoft's Alpine and Ubuntu Chiseled images ship without ICU (their extra variants add it), and the .NET 10 Native AOT templates enable it: dotnet new console --aot, dotnet new worker --aot and dotnet new webapiaot all set <InvariantGlobalization>true</InvariantGlobalization>. A service that tokenizes correctly in development can silently stop stripping accents when deployed in one of these configurations.

Reproduction

The reproduction below uses the vocab.txt from sentence-transformers/all-MiniLM-L6-v2.

using System.Text;
using Microsoft.ML.Tokenizers;

var vocabPath = args[0];
var vocab = File.ReadAllLines(vocabPath);
var tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

Console.WriteLine($"Normalize(FormD) of \"\\u00e9\": {"\u00e9".Normalize(NormalizationForm.FormD).Length} code unit(s)");
var ids = tokenizer.EncodeToIds("Café crème brûlée in São Paulo, naïve résumé");
Console.WriteLine(string.Join(' ', ids.Select(id => vocab[id])));

Run it once as built, and once in invariant mode:

dotnet run -- path/to/vocab.txt
dotnet run -p:InvariantGlobalization=true -- path/to/vocab.txt

Under ICU:

Normalize(FormD) of "\u00e9": 2 code unit(s)
[CLS] cafe cr ##eme br ##ule ##e in sao paulo , naive resume [SEP]

Under invariant globalization:

Normalize(FormD) of "\u00e9": 1 code unit(s)
[CLS] [UNK] [UNK] [UNK] in [UNK] paulo , [UNK] [UNK] [SEP]

The ICU output matches Hugging Face's BertTokenizer exactly. The invariant output is identical to running with RemoveNonSpacingMarks = false.

Only precomposed input is affected; input that is already decomposed is handled correctly because its combining marks are present to be dropped.

Expected behaviour

Either of these would remove the silent failure:

  1. Strip accents without relying on string.Normalize, for example by carrying the decomposition data the option needs, so it behaves the same in every globalization mode; or
  2. Fail loudly: throw from BertTokenizer.Create when RemoveNonSpacingMarks = true and decomposition is unavailable, with a message naming the fix (<InvariantGlobalization>false</InvariantGlobalization>, or an ICU-carrying image such as a chiseled -extra variant).

The check for (2) is one line at construction: "\u00e9".Normalize(NormalizationForm.FormD).Length == 2.

Impact

In my test with all-MiniLM-L6-v2, mean pooling, and cosine similarity against the ICU tokenization, the embeddings of affected text diverge substantially:

input ICU invariant
Café crème brûlée in São Paulo, naïve résumé 1.000000 0.333027
a French sentence 1.000000 0.545866
a Spanish sentence 1.000000 0.502484
a German sentence 1.000000 0.572320
accented capitals (ÉCOLE ÉLÉMENTAIRE) 1.000000 0.089375
plain ASCII 1.000000 1.000000
Versions
  • Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, identical output
  • .NET 10.0.12
Related

#7724 and #7725 report other BertTokenizer differences from Hugging Face's tokenizer. This one is independent of them: the output is correct under ICU and wrong only because of the globalization mode.

Lenguaje dominante
C#
Estrellas
9.4k
Forks
2k
Merge medio
2 d 18 h
PR fusionados (30 d)
64

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de dotnet/machinelearning

Todos los issues de dotnet/machinelearning

Issues similares

Más issues de C#

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.