BertTokenizer merges words separated only by `\n`, `\t` or `\r`

Start in src/Microsoft.ML.Tokenizers/Normalizer/BertNormalizer.cs and inspect how UnicodeCategory.Control characters are handled before whitespace pre-tokenization. Reproduce the issue with the provided one\nline, one\tline, and note\nbook examples; done means these characters are treated as whitespace and the outputs match the expected tokenization.

Description

BertTokenizer deletes \n, \t and \r rather than treating them as whitespace, so the words on either side are joined into one word before tokenization. What that yields depends on the vocabulary: one\nline becomes one ##line, while note\nbook becomes the single token notebook. Either way it changes the tokens, and therefore the embeddings, for any text with a line break or tab that has no adjacent space, which is common in real documents.

Hugging Face's BERT basic tokenizer treats these three characters as whitespace: _is_control explicitly excludes them, and _is_whitespace includes them. That is not specific to HF — it comes from the original BERT release, whose tokenization.py returns False from _is_control for \t, \n and \r, commented "These are technically control characters but we count them as whitespace characters". BertTokenizer's own remarks state that its implementation "is based on the original Bert implementation in the Hugging Face Transformers library".

If this is deliberate — a decision to treat Control uniformly rather than follow the _is_whitespace exception — please share the reasoning, and I'll close this.

Versions

• Microsoft.ML.Tokenizers 2.0.0 and 3.0.0-preview.26457.2, with identical results
• .NET 10.0.12, macOS arm64
• Vocabulary: sentence-transformers/all-MiniLM-L6-v2 vocab.txt (uncased BERT, 30,522 tokens)
• Reference: Hugging Face tokenizers 0.21.1, BertWordPieceTokenizer(vocab, lowercase=True)
• Cosine under "Impact": onnxruntime 1.22.0, mean pooling over the token ids

Reproduce

using Microsoft.ML.Tokenizers;

var vocab = File.ReadAllLines("vocab.txt");
var tokenizer = BertTokenizer.Create("vocab.txt", new BertOptions());
foreach (var input in new[] { "one\nline", "one\tline", "one \nline", "note\nbook" })
{
    var tokens = tokenizer.EncodeToIds(input).Select(id => vocab[id]);
    Console.WriteLine($"{input.Replace("\n", "\\n").Replace("\t", "\\t")} -> {string.Join(' ', tokens)}");
}

Expected vs actual

No BertOptions setting preserves BERT basic-tokenization semantics while correcting this behavior. Setting ApplyBasicTokenization = false bypasses the faulty normalizer, but also disables the rest of BERT basic tokenization.

Cause

BertNormalizer.Normalize (src/Microsoft.ML.Tokenizers/Normalizer/BertNormalizer.cs) skips every character whose category is UnicodeCategory.Control:

if (category == UnicodeCategory.Control)
{
    i += inc;
    continue;
}

\t, \n and \r are all Control, so they are removed before the pre-tokenizer splits on whitespace. A fix is to emit a space for those three characters before that check, as HF does:

if (c is '\t' or '\n' or '\r')
{
    AddChar(ref buffer, ref index, ' ');
    continue;
}

Impact

Embedding all-MiniLM-L6-v2 through ONNX Runtime with mean pooling, the input " line one\n\tline two " scores cosine 0.936 against the same text tokenized by HF tokenizers. Callers can work around it by replacing \t, \n and \r with spaces before encoding.