BertTokenizer: `RemoveNonSpacingMarks = true` fails to strip accents from precomposed characters under invariant globalization

Aberta
  • Algo errado?
#7,728 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Bug
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
csharp
Domínio
machine-learning

Direção de pesquisa

Start at BertTokenizer.Create and the RemoveNonSpacingMarks handling, then run the supplied reproduction once with ICU and once with InvariantGlobalization=true. The issue is done when precomposed accents are handled consistently across both modes, or creation fails clearly when decomposition is unavailable; add coverage for the reported inputs and configuration.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

area-Tokenizers green

BertOptions.RemoveNonSpacingMarks = true is the setting that makes BertTokenizer strip accents, as uncased BERT vocabularies expect: café becomes cafe. Accent stripping works by decomposing the text to Unicode Form D and dropping the non-spacing marks. Under globalization-invariant mode, string.Normalize returns its input unchanged — that is documented behaviour — so there are no marks to drop. Precomposed accented characters remain intact; for vocabularies such as all-MiniLM-L6-v2, this can cause affected words to miss their expected WordPiece decomposition and become [UNK].

Nothing signals this. BertTokenizer.Create succeeds, encoding succeeds, and for precomposed text the output is identical to having left the option off.

Invariant globalization is common in size-optimised .NET deployments. Microsoft's Alpine and Ubuntu Chiseled images ship without ICU (their extra variants add it), and the .NET 10 Native AOT templates enable it: dotnet new console --aot, dotnet new worker --aot and dotnet new webapiaot all set <InvariantGlobalization>true</InvariantGlobalization>. A service that tokenizes correctly in development can silently stop stripping accents when deployed in one of these configurations.

Reproduction

The reproduction below uses the vocab.txt from sentence-transformers/all-MiniLM-L6-v2.

using System.Text;
using Microsoft.ML.Tokenizers;

var vocabPath = args[0];
var vocab = File.ReadAllLines(vocabPath);
var tokenizer = BertTokenizer.Create(vocabPath, new BertOptions { RemoveNonSpacingMarks = true });

Console.WriteLine($"Normalize(FormD) of \"\\u00e9\": {"\u00e9".Normalize(NormalizationForm.FormD).Length} code unit(s)");
var ids = tokenizer.EncodeToIds("Café crème brûlée in São Paulo, naïve résumé");
Console.WriteLine(string.Join(' ', ids.Select(id => vocab[id])));

Run it once as built, and once in invariant mode:

dotnet run -- path/to/vocab.txt
dotnet run -p:InvariantGlobalization=true -- path/to/vocab.txt

Under ICU:

Normalize(FormD) of "\u00e9": 2 code unit(s)
[CLS] cafe cr ##eme br ##ule ##e in sao paulo , naive resume [SEP]

Under invariant globalization:

Normalize(FormD) of "\u00e9": 1 code unit(s)
[CLS] [UNK] [UNK] [UNK] in [UNK] paulo , [UNK] [UNK] [SEP]

The ICU output matches Hugging Face's BertTokenizer exactly. The invariant output is identical to running with RemoveNonSpacingMarks = false.

Only precomposed input is affected; input that is already decomposed is handled correctly because its combining marks are present to be dropped.

Expected behaviour

Either of these would remove the silent failure:

  1. Strip accents without relying on string.Normalize, for example by carrying the decomposition data the option needs, so it behaves the same in every globalization mode; or
  2. Fail loudly: throw from BertTokenizer.Create when RemoveNonSpacingMarks = true and decomposition is unavailable, with a message naming the fix (<InvariantGlobalization>false</InvariantGlobalization>, or an ICU-carrying image such as a chiseled -extra variant).

The check for (2) is one line at construction: "\u00e9".Normalize(NormalizationForm.FormD).Length == 2.

Impact

In my test with all-MiniLM-L6-v2, mean pooling, and cosine similarity against the ICU tokenization, the embeddings of affected text diverge substantially:

input ICU invariant
Café crème brûlée in São Paulo, naïve résumé 1.000000 0.333027
a French sentence 1.000000 0.545866
a Spanish sentence 1.000000 0.502484
a German sentence 1.000000 0.572320
accented capitals (ÉCOLE ÉLÉMENTAIRE) 1.000000 0.089375
plain ASCII 1.000000 1.000000
Versions
  • Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, identical output
  • .NET 10.0.12
Related

#7724 and #7725 report other BertTokenizer differences from Hugging Face's tokenizer. This one is independent of them: the output is correct under ICU and wrong only because of the globalization mode.

Linguagem predominante
C#
Estrelas
9.4k
Forks
2k
Merge médio
2d 18h
PRs com merge (30d)
64

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de dotnet/machinelearning

Todas as issues de dotnet/machinelearning

Issues semelhantes

Mais issues de C#

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.