Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Lane 2 · kotlin-core] tokenizer.json parity — apply the normalizer block (Replace, Prepend, Sequence, NFC) before pre-tokenization

Open
  • Is something wrong?
#1,325 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
43/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
kotlin
Domain
backend, data

Research direction

Start with SpecialTokenSplitter.kt and the normalizer handling in SentencePieceTokenizer.kt#L382-L391; the issue names both tokenizer factories to update and the existing source sets for platform NFC code. First check the Lane 1 survey for the NFC strategy, then implement the normalizer plumbing while it is open. Run the listed jvmTest and linuxX64Test; done means the three specified normalizers pass, unknown types fail with their names, and the ModernBERT fixture ends with id 3974.

Written by the indexing model from the issue text.

Description

coding size:m skill:kotlin-core sub-issue

Sub-issue of #1321 (HF tokenizer.json parity for BPE encoders).

Lane: 2 · Kotlin core
Skill needed: Kotlin Multiplatform (expect/actual or a pure-Kotlin table), Unicode normalization basics. No tensor knowledge.
Size: m (1–2 days; most of it is the NFC implementation and its tests)
Blocked by: the Lane 1 survey (it decides the NFC strategy). Start with the Replace/Sequence plumbing while it is open.

What to do

  1. Add a small Normalizer step to sk.ainet.io.tokenizer that is built from tokenizer.json#normalizer and applied to each non-special segment before pre-tokenization (the reference applies it before added-token matching only for tokens with normalized: true; applying it to the segments between atomic matches is the observable behaviour for the files in the parent).
  2. Support, in this order: null (no-op), Replace (pattern.String → content; pattern.Regex can throw UnsupportedTokenizerException with the pattern in the message), Prepend, Sequence (compose), NFC. Everything else: throw UnsupportedTokenizerException naming the type — do not silently skip, that is the current bug.
  3. Implement NFC the way Lane 1 recommended. If that is expect/actual: JVM via java.text.Normalizer, JS/Wasm via normalize("NFC"), Kotlin/Native passthrough with a documented warning in the KDoc. Put the per-target code in the existing jvmMain / jsMain / wasmJsMain / native64Main source sets of skainet-io-core.
  4. Wire it into QwenByteLevelBpeTokenizer.fromTokenizerJson and SentencePieceTokenizer.fromTokenizerJson; the existing Prepend sniffing at SentencePieceTokenizer.kt#L382-L391 should become a consumer of the new step rather than a second parser.
  5. Tests in commonTest: Replace, Sequence, and NFC on "e\u0301" → "é" (JVM at least; mark the native expectation explicitly). Run ./gradlew :skainet-io:skainet-io-core:jvmTest :skainet-io:skainet-io-core:linuxX64Test.

Acceptance

  • ModernBERT file: the third fixture text of #1321 (ending in decomposed e + U+0301) encodes identically to the reference — last id 3974, not 299, 136, 212
  • Unknown normalizer types fail loudly with the type name in the message
  • Tests green on jvmTest and linuxX64Test
  • Result reported back on #1321

Notes

Sibling file to copy the shape from: SpecialTokenSplitter.kt is a small decorator built by the factory from a JSON block — the normalizer is the same pattern one step earlier in the pipeline. Keep the implementation out of QwenByteLevelBpeTokenizer itself so the Metaspace lane can reuse it.

Dominant language
Kotlin
Stars
52
Forks
15
Avg merge
1d 15h
Merged PRs (30d)
36

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SKaiNET-developers/SKaiNET

All issues in SKaiNET-developers/SKaiNET

Similar issues

More Kotlin issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.