Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Lane 2 · kotlin-core] tokenizer.json parity — match non-special added_tokens atomically and count them in vocabSize

Closed
  • Is something wrong?
#1,324 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
kotlin
Domain
tooling

Research direction

Read the added-token loop in QwenByteLevelBpeTokenizer.kt and TokenizerFactory.kt, then inspect the related tests in TokenizerFactoryDispatchTest.kt. Add a synthetic case where a non-special token would otherwise split, and check that it encodes atomically and vocabSize includes its id. Run ./gradlew :skainet-io:skainet-io-core:jvmTest; done means the test passes and the changelog records the vocabSize change.

Written by the indexing model from the issue text.

Description

coding good first issue size:s skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin. Helpful to have read how tokenizers matches added tokens (one page of docs, linked below). No tensor knowledge.
Size: s (a few hours)
Blocked by: nothing (independent of the merges lane; both touch the same file, so rebase once)

What to do

  1. Read how the reference matches added tokens: every entry of added_tokens is matched atomically, before pre-tokenization, regardless of special. The special flag only decides whether the token is dropped by decode(skip_special_tokens=True). See the AddedVocabulary section of the tokenizers docs: https://huggingface.co/docs/tokenizers/api/added-tokens
  2. In QwenByteLevelBpeTokenizer.kt#L257-L267 the loop skips entries with "special": false. Register all entries in the atomic-match map. Keep a separate Set<Int> of the ids that are marked special so a later change can implement skip_special_tokens; nothing else needs that set today.
  3. Make the same change in TokenizerFactory.kt#L155-L169 for the Unigram path (wrapSentencePieceWithSpecialsFromJson).
  4. Added tokens may have ids above model.vocab (ModernBERT: [UNK]=50280 … and the whitespace runs up to 50367). Extend the tokens array so vocabSize equals the reference get_vocab_size(with_added_tokens=True) — 50368 for ModernBERT instead of today's 50280 — and so decode can return their text.
  5. Tests, in TokenizerFactoryDispatchTest.kt: a synthetic file with one special: false added token whose content would otherwise BPE-split; assert it encodes to its id and that vocabSize counts it. Run ./gradlew :skainet-io:skainet-io-core:jvmTest.

Acceptance

  • Synthetic test: non-special added token encodes atomically; vocabSize includes ids above model.vocab
  • With the ModernBERT file (tokenizer/tokenizer.json) and merges loading (sibling lane), " leading" encodes to [50275, 16378] (reference: [' ', 'leading']) instead of [245, 4283, …], and vocabSize == 50368
  • CHANGELOG.md notes the vocabSize behaviour change
  • Result reported back on #1321

Notes

The flags single_word, lstrip, rstrip, normalized exist on every entry. In the two Laya files they are all false except normalized: true on the non-special ModernBERT entries (and lstrip: true on one special token). Implement plain longest-match as the existing special-token code does; if a file you care about sets single_word or lstrip/rstrip, open a separate issue rather than growing this one. #912 asks for the same vocabSize fix on the SentencePiece side — closing that part here is fine, say so in the PR.

Dominant language
Kotlin
Stars
52
Forks
15
Avg merge
1d 4h
Merged PRs (30d)
37

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SKaiNET-developers/SKaiNET

All issues in SKaiNET-developers/SKaiNET

Similar issues

More Kotlin issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.